kamikazechaser/cic-internal-integration
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

WIP add details on daemons and tools

Browse Source
This commit is contained in:
nolash 2021-04-27 13:09:13 +02:00
parent 601079433b
commit d93d52d20a
Signed by untrusted user who does not match committer: lash
GPG Key ID: 21D2E7BB88C2A746
1 changed files with 187 additions and 92 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

279
apps/cic-eth/doc/texinfo/overview.texi
View File

@ -12,7 +12,7 @@ The current implementation is in its entirety based on the EVM network and with
@subsection Dependencies @subsection Dependencies
This application is written in Python 3.8. It is tightly coupled with @code{python-celery}, which provides the task worker ecosystem, and @code{SQLAlchemy} which provides useful abstractions for persistent storage though SQL. This application is written in Python 3.8. It is tightly coupled with @code{python-celery}, which provides the task worker ecosystem. It also uses @code{SQLAlchemy} which provides useful abstractions for persistent storage though SQL, and @code{alembic} for database schema migrations.
These is currently also a coupling with @code{Redis}, which is used as message broker for @code{python-celery}. This coupling may be relaxed in the future to allow other key-value pubsub solutions instead. @code{Redis} is also explicitly used by some CLI tools to retrieve results from command execution. These is currently also a coupling with @code{Redis}, which is used as message broker for @code{python-celery}. This coupling may be relaxed in the future to allow other key-value pubsub solutions instead. @code{Redis} is also explicitly used by some CLI tools to retrieve results from command execution.
@ -54,6 +54,112 @@ Called by newly created accounts to receive initial token balance
The closely related component @code{cic-eth-registry} facilitates lookups of resources on the blockchain network. In its current state it resolves tokens by symbol or address, and contracts by common-name identifiers. The closely related component @code{cic-eth-registry} facilitates lookups of resources on the blockchain network. In its current state it resolves tokens by symbol or address, and contracts by common-name identifiers.
@section Configuration
(refer to @code{cic-base} for a general overview of the config pipeline)
Configuration parameters are grouped by configuration filename.
@subsection cic
@table @var
@item registry_address
Ethereum address of the @var{ContractRegistry} contract
@item chain_spec
String representation of the connected blockchain according to the @var{chainlib} @var{ChainSpec} format.
@item tx_retry_delay
Minimum time in seconds to wait before retrying a transaction
@item trust_address
Comma-separated list of one or more ethereum addresses regarded as trusted for describing other resources, Used by @var{cic-eth-registry} in the context of the @var{AddressDeclarator}.
@item defalt_token_symbol
Fallback token to operate on when no other context is given.
@item health_modules
Comma-separated list of methods to execute liveness tests against. (see ...)
@item run_dir
Directory to use for session-scoped variables for @var{cic-eth} daemon parent processes.
@end table
@subsection celery
@table @var
@item broker_url
Message broker URL
@item result_url
Result backend URL
@item debug
Boolean value. If set, the amount of available context for a task in the result backend will be maximized@footnote{THis is a @emph{required} setting for the task graph documenter to enabled it to display task names in the graph}.
@end table
@subsection database
See ref cic-base when ready
@subsection eth
@table @var
@item provider
Address of default RPC endpoint for transactions and state queries.
@item gas_gifter_minimum_balance
The minimum gas balance that must be held by the @code{GAS GIFTER} token before the queue processing shuts down@footnote{You should really make sure that this threshold is never hit}
@end table
@subsection redis
Defines connection to the redis server used outside of the context of @var{celery}. This is usually the same server, but should be a different db.
@table @var
@item host
Redis hostname
@item port
Redis port
@item db
Redis db
@end table
@subsection signer
Parameters
@table @var
@item socket_path
The connection string for the signer JSON-RPC service.@footnote{The @var{crypto-dev-signer} supports UNIX socket or a HTTP(S) connections}
@item secret
If set, this password is used to add obfuscation on top of the encryption already applied by the signer for the keystore.
@end table
@subsection ssl
Certificate information for https api callbacks.
@table @var
@item enable_client
Boolean value. If set, client certificate will be used to authenticate the callback request.
@item cert_file
Client certificate file in PEM or DER format
@item key_file
Client key file in PEM or DER format
@item password
Password for unlocking the client key
@item ca_file
Certificate authority bundle, to verify the certificate sent by the callback server.
@end table
@subsection syncer
@table @var
@item loop_interval
Seconds to pause before each execution of the @var{chainsyncer} poll loop.
@end table
@section Interacting with the system @section Interacting with the system
The API to the @code{cic-eth} component is a proxy for executing @emph{chains of Celery tasks}. The tasks that compose individual chains are documented in appendix (?), which also describes a CLI tool that can generate graph representationso of them. The API to the @code{cic-eth} component is a proxy for executing @emph{chains of Celery tasks}. The tasks that compose individual chains are documented in appendix (?), which also describes a CLI tool that can generate graph representationso of them.
@ -278,11 +384,11 @@ Executes, in order, Celery tasks defined in the configuration variable @var{TASK
The callbacks will receive the following arguments The callbacks will receive the following arguments
@enumerate @enumerate
@item result @item @strong{result}
A complex representation of the transaction (see section ?) A complex representation of the transaction (see section ?)
@item transfertype @item @strong{transfertype}
A string describing the type of transaction detected@footnote{See appendix ? for an overview of possible values} A string describing the type of transaction found@footnote{See appendix ? for an overview of possible values}
@item status @item @strong{status}
0 if contract code executed successfully. Any other value is an error@footnote{The values 1-1024 are reserved for system specific errors. In the current implementation only a general error state with value 1 is defined. See appendix ?.} 0 if contract code executed successfully. Any other value is an error@footnote{The values 1-1024 are reserved for system specific errors. In the current implementation only a general error state with value 1 is defined. See appendix ?.}
@end enumerate @end enumerate
@ -296,109 +402,98 @@ If a valid transfer authorization request has been made, a token @emph{allowance
If the transaction is a token conversion, @emph{and} there is a pending transfer registered for the conversion, the corresponding token transfer transaction will be executed. Not currently implemented If the transaction is a token conversion, @emph{and} there is a pending transfer registered for the conversion, the corresponding token transfer transaction will be executed. Not currently implemented
@section Configuration @section Services
(refer to @code{cic-base} for a general overview of the config pipeline) There are four daemons that together orchestrate all of the aforementioned recipes. This section will provide a high level description of them.
Configuration parameters are grouped by configuration filename. Each of them have their own set of command line flags. These are available in the CLI help text provided by @kbd{-h} @kbd{--help} and are not recited here.
Daemon executable scripts are located in the @file{cic_eth.runnable.daemons} package. If @var{cic-eth} is installed as a python package, they are installed as executables in @var{PATH}.
@subsection cic @subsection tasker
This is the heart of the custodial system. Tasker is the parent process for the celery workers executing all tasks interacting with and changing the state of the queue and the chain. It is also the only service that interfaces with the signer/keystore.
The other @var{cic-eth} daemons all interface with this component, along with any client adapter bridging an end-user gateway (e.g. @var{cic-ussd}). However, the service itself does not have to be actively running for the other services to run; @var{Celery} handles queueing up the incoming tasks until the @var{tasker} comes back online.@footnote{Whereas this is true, there is currently no fail-safe implemented to handles the event of task backlog overflow in Celery. Furthermore, no targeted testing has yet been performed to asses the stability of the system over time if a sudden, sustained surge of resumed task executions occurs. It may be advisable to suspend activity that adds new queue items to the system if volume is high and/or the @var{cic-eth} outage endures. However, there is no panacea for this condition, as every usage scenario is different}
The tasker has a set of pre-requisites that must be fulfilled before it will start
@itemize
@item It must be given a valid @var{ContractRegistry} address, which must include valid references to all contracts specified in @strong{[point to section about contract deps]})
@item The gas gifter balance must be above the minimum threshold (See "eth" section in configurations).
@item There must be a valid alembic migration record in the storage database
@item The redis backend must be reachable and writable
@item There must be a reachable JSON-RPC server at the other end of the signer socket path (see "signer" section in configurations)
@end itemize
@subsection tracker
Implements the @var{chainsyncer}, and registers the filters described in @strong{[ref transaction monitoring]} to be executed for every transaction. It consumes the appropriate @var{TASKS_TRANSFER_CALLBACKS} configuration setting to add externally defined filters at without having to change the daemon code.
The @var{tracker} has the same requisities for the @var{ContractRegistry} as the @var{tasker}.
@strong{Important! Guarantees of filter executions has some caveats. Refer to the @var{chainsyncer} documentation for more details.}
@subsection dispatcher
Uses the @code{get_upcoming_tx} method call from @var{chainqueue} to receive batches of queued transactions that are ready to send to the blockchain node. Every batch will only contain a single transaction by any one address, which will be the transaction with the next nonce not previously seen by the network. There is no limit currently set to how many transactions that will be included in a single batch.
@subsection retrier
The responsibility of the @var{retrier} is to re-queue transactions that failed to be sent to the blockchain node, as well as create @emph{replacements} for transactions whose processing by the network has been delayed. @strong{[refer transaction obolestion]}.
It is in turn the responsiblity of the @var{dispatcher} to send these (re-)queued transactions to the blockchain node.
@section Tools
A collection of CLI tools have been provided to help with diagnostics and other administrative tasks. These use the same configuration infrastructure as the daemons.
Tool scripts are located in the @file{cic_eth.runnable} package. If @var{cic-eth} is installed as a python package, they are installed as executables in @var{PATH}.
@subsection info (cic-eth-info)
Returns self-explanatory metadata for the blockchain network, and optionally an address.
@subsection inspect (cic-eth-inspect)
Returns information about a specific resource related to the tranasaction queue. The results returned depend on the type of the argument.
@table @var @table @var
@item registry_address @item lock
Ethereum address of the @var{ContractRegistry} contract If the argument is the literal string @kbd{lock}, it will list all active lock settings currently in effect. (see @strong{[reference locking section]})
@item chain_spec @item <address>
String representation of the connected blockchain according to the @var{chainlib} @var{ChainSpec} format. If the argument is a 0x-prefixed hex string of 42 characters, it returns all transactions where the specified address is a sender or recipient@footnote{If the address is the gas gifter or the accounts index writer, this may be a @emph{lot} of transactions. Use with care!}
@item tx_retry_delay @item <tx_hash>
Minimum time in seconds to wait before retrying a transaction If the argument is a 0x-prefixed hex string of 66 characters, it returns data from the custodial queueing system aswell as the network for a single transaction whose hash matches the input. Fails if the transaction does not exist in the queue
@item trust_address @item <code>
Comma-separated list of one or more ethereum addresses regarded as trusted for describing other resources, Used by @var{cic-eth-registry} in the context of the @var{AddressDeclarator}. If the argument is a 0x-prefixed hex string longer than 66 bytes, the argument will be interpreted as raw RLP serialized transaction data, and attempt to match this with an entry in the queue. If a match is found, the result is the same as for @var{<tx_hash>}
@item defalt_token_symbol
Fallback token to operate on when no other context is given.
@item health_modules
Comma-separated list of methods to execute liveness tests against. (see ...)
@item run_dir
Directory to use for session-scoped variables for @var{cic-eth} daemon parent processes.
@end table @end table
@subsection celery @subsection create (cic-eth-create)
@table @var Create a new account, optionally registering the account in the accounts registry, and optionally receiving the newly created address through a redis subscription.
@item broker_url
Message broker URL
@item result_url
Result backend URL
@item debug
Boolean value. If set, the amount of available context for a task in the result backend will be maximized@footnote{THis is a @emph{required} setting for the task graph documenter to enabled it to display task names in the graph}.
@end table
@subsection transfer (cic-eth-transfer)
@subsection database Execute a token transfer on behalf of a custodial account.
See ref cic-base when ready @subsection tag (cic-eth-tag)
Associate an account address with a string identifier. See @strong{[reference SYSTEM ACCOUNTS]}.
@subsection eth @subsection ctrl (cic-eth-ctrl)
@table @var Set or reset lock bits, globally or per account address.
@item provider
Address of default RPC endpoint for transactions and state queries.
@item gas_gifter_minimum_balance
The minimum gas balance that must be held by the @code{GAS GIFTER} token before the queue processing shuts down@footnote{You should really make sure that this threshold is never hit}
@end table
@subsection resend (cic-eth-resend)
@subsection redis Resend a transaction. This can either be done "in-place," which means increasing the gas price and re-queueing@footnote{this is the same thing that the retrier does}. It can also be used to @emph{clone} a transaction, which obviously will duplicate the effect of the cloned transaction on the blockchain network.
Defines connection to the redis server used outside of the context of @var{celery}. This is usually the same server, but should be a different db.
@table @var
@item host
Redis hostname
@item port
Redis port
@item db
Redis db
@end table
@subsection signer
Parameters
@table @var
@item socket_path
The connection string for the signer JSON-RPC service.@footnote{The @var{crypto-dev-signer} supports UNIX socket or a HTTP(S) connections}
@item secret
If set, this password is used to add obfuscation on top of the encryption already applied by the signer for the keystore.
@end table
@subsection ssl
Certificate information for https api callbacks.
@table @var
@item enable_client
Boolean value. If set, client certificate will be used to authenticate the callback request.
@item cert_file
Client certificate file in PEM or DER format
@item key_file
Client key file in PEM or DER format
@item password
Password for unlocking the client key
@item ca_file
Certificate authority bundle, to verify the certificate sent by the callback server.
@end table
@subsection syncer
@table @var
@item loop_interval
Seconds to pause before each execution of the @var{chainsyncer} poll loop.
@end table