WIP add details on daemons and tools
This commit is contained in:
nolash
parent
601079433b
commit
d93d52d20a
GPG Key ID:
21D2E7BB88C2A746
|
|
@ -12,7 +12,7 @@ The current implementation is in its entirety based on the EVM network and with
|
|||
|
||||
@subsection Dependencies
|
||||
|
||||
This application is written in Python 3.8. It is tightly coupled with @code{python-celery}, which provides the task worker ecosystem, and @code{SQLAlchemy} which provides useful abstractions for persistent storage though SQL.
|
||||
This application is written in Python 3.8. It is tightly coupled with @code{python-celery}, which provides the task worker ecosystem. It also uses @code{SQLAlchemy} which provides useful abstractions for persistent storage though SQL, and @code{alembic} for database schema migrations.
|
||||
|
||||
These is currently also a coupling with @code{Redis}, which is used as message broker for @code{python-celery}. This coupling may be relaxed in the future to allow other key-value pubsub solutions instead. @code{Redis} is also explicitly used by some CLI tools to retrieve results from command execution.
|
||||
|
||||
|
|
@ -54,6 +54,112 @@ Called by newly created accounts to receive initial token balance
|
|||
The closely related component @code{cic-eth-registry} facilitates lookups of resources on the blockchain network. In its current state it resolves tokens by symbol or address, and contracts by common-name identifiers.
|
||||
|
||||
|
||||
@section Configuration
|
||||
|
||||
(refer to @code{cic-base} for a general overview of the config pipeline)
|
||||
|
||||
Configuration parameters are grouped by configuration filename.
|
||||
|
||||
|
||||
@subsection cic
|
||||
|
||||
@table @var
|
||||
@item registry_address
|
||||
Ethereum address of the @var{ContractRegistry} contract
|
||||
@item chain_spec
|
||||
String representation of the connected blockchain according to the @var{chainlib} @var{ChainSpec} format.
|
||||
@item tx_retry_delay
|
||||
Minimum time in seconds to wait before retrying a transaction
|
||||
@item trust_address
|
||||
Comma-separated list of one or more ethereum addresses regarded as trusted for describing other resources, Used by @var{cic-eth-registry} in the context of the @var{AddressDeclarator}.
|
||||
@item defalt_token_symbol
|
||||
Fallback token to operate on when no other context is given.
|
||||
@item health_modules
|
||||
Comma-separated list of methods to execute liveness tests against. (see ...)
|
||||
@item run_dir
|
||||
Directory to use for session-scoped variables for @var{cic-eth} daemon parent processes.
|
||||
@end table
|
||||
|
||||
|
||||
@subsection celery
|
||||
|
||||
@table @var
|
||||
@item broker_url
|
||||
Message broker URL
|
||||
@item result_url
|
||||
Result backend URL
|
||||
@item debug
|
||||
Boolean value. If set, the amount of available context for a task in the result backend will be maximized@footnote{THis is a @emph{required} setting for the task graph documenter to enabled it to display task names in the graph}.
|
||||
@end table
|
||||
|
||||
|
||||
@subsection database
|
||||
|
||||
See ref cic-base when ready
|
||||
|
||||
|
||||
@subsection eth
|
||||
|
||||
@table @var
|
||||
@item provider
|
||||
Address of default RPC endpoint for transactions and state queries.
|
||||
@item gas_gifter_minimum_balance
|
||||
The minimum gas balance that must be held by the @code{GAS GIFTER} token before the queue processing shuts down@footnote{You should really make sure that this threshold is never hit}
|
||||
@end table
|
||||
|
||||
|
||||
@subsection redis
|
||||
|
||||
Defines connection to the redis server used outside of the context of @var{celery}. This is usually the same server, but should be a different db.
|
||||
|
||||
@table @var
|
||||
@item host
|
||||
Redis hostname
|
||||
@item port
|
||||
Redis port
|
||||
@item db
|
||||
Redis db
|
||||
@end table
|
||||
|
||||
|
||||
@subsection signer
|
||||
|
||||
Parameters
|
||||
|
||||
@table @var
|
||||
@item socket_path
|
||||
The connection string for the signer JSON-RPC service.@footnote{The @var{crypto-dev-signer} supports UNIX socket or a HTTP(S) connections}
|
||||
@item secret
|
||||
If set, this password is used to add obfuscation on top of the encryption already applied by the signer for the keystore.
|
||||
@end table
|
||||
|
||||
|
||||
|
||||
@subsection ssl
|
||||
|
||||
Certificate information for https api callbacks.
|
||||
|
||||
@table @var
|
||||
@item enable_client
|
||||
Boolean value. If set, client certificate will be used to authenticate the callback request.
|
||||
@item cert_file
|
||||
Client certificate file in PEM or DER format
|
||||
@item key_file
|
||||
Client key file in PEM or DER format
|
||||
@item password
|
||||
Password for unlocking the client key
|
||||
@item ca_file
|
||||
Certificate authority bundle, to verify the certificate sent by the callback server.
|
||||
@end table
|
||||
|
||||
|
||||
@subsection syncer
|
||||
|
||||
@table @var
|
||||
@item loop_interval
|
||||
Seconds to pause before each execution of the @var{chainsyncer} poll loop.
|
||||
@end table
|
||||
|
||||
@section Interacting with the system
|
||||
|
||||
The API to the @code{cic-eth} component is a proxy for executing @emph{chains of Celery tasks}. The tasks that compose individual chains are documented in appendix (?), which also describes a CLI tool that can generate graph representationso of them.
|
||||
|
|
@ -278,11 +384,11 @@ Executes, in order, Celery tasks defined in the configuration variable @var{TASK
|
|||
The callbacks will receive the following arguments
|
||||
|
||||
@enumerate
|
||||
@item result
|
||||
@item @strong{result}
|
||||
A complex representation of the transaction (see section ?)
|
||||
@item transfertype
|
||||
A string describing the type of transaction detected@footnote{See appendix ? for an overview of possible values}
|
||||
@item status
|
||||
@item @strong{transfertype}
|
||||
A string describing the type of transaction found@footnote{See appendix ? for an overview of possible values}
|
||||
@item @strong{status}
|
||||
0 if contract code executed successfully. Any other value is an error@footnote{The values 1-1024 are reserved for system specific errors. In the current implementation only a general error state with value 1 is defined. See appendix ?.}
|
||||
@end enumerate
|
||||
|
||||
|
|
@ -296,109 +402,98 @@ If a valid transfer authorization request has been made, a token @emph{allowance
|
|||
If the transaction is a token conversion, @emph{and} there is a pending transfer registered for the conversion, the corresponding token transfer transaction will be executed. Not currently implemented
|
||||
|
||||
|
||||
@section Configuration
|
||||
@section Services
|
||||
|
||||
(refer to @code{cic-base} for a general overview of the config pipeline)
|
||||
There are four daemons that together orchestrate all of the aforementioned recipes. This section will provide a high level description of them.
|
||||
|
||||
Configuration parameters are grouped by configuration filename.
|
||||
Each of them have their own set of command line flags. These are available in the CLI help text provided by @kbd{-h} @kbd{--help} and are not recited here.
|
||||
|
||||
Daemon executable scripts are located in the @file{cic_eth.runnable.daemons} package. If @var{cic-eth} is installed as a python package, they are installed as executables in @var{PATH}.
|
||||
|
||||
|
||||
@subsection cic
|
||||
@subsection tasker
|
||||
|
||||
This is the heart of the custodial system. Tasker is the parent process for the celery workers executing all tasks interacting with and changing the state of the queue and the chain. It is also the only service that interfaces with the signer/keystore.
|
||||
|
||||
The other @var{cic-eth} daemons all interface with this component, along with any client adapter bridging an end-user gateway (e.g. @var{cic-ussd}). However, the service itself does not have to be actively running for the other services to run; @var{Celery} handles queueing up the incoming tasks until the @var{tasker} comes back online.@footnote{Whereas this is true, there is currently no fail-safe implemented to handles the event of task backlog overflow in Celery. Furthermore, no targeted testing has yet been performed to asses the stability of the system over time if a sudden, sustained surge of resumed task executions occurs. It may be advisable to suspend activity that adds new queue items to the system if volume is high and/or the @var{cic-eth} outage endures. However, there is no panacea for this condition, as every usage scenario is different}
|
||||
|
||||
The tasker has a set of pre-requisites that must be fulfilled before it will start
|
||||
|
||||
@itemize
|
||||
@item It must be given a valid @var{ContractRegistry} address, which must include valid references to all contracts specified in @strong{[point to section about contract deps]})
|
||||
@item The gas gifter balance must be above the minimum threshold (See "eth" section in configurations).
|
||||
@item There must be a valid alembic migration record in the storage database
|
||||
@item The redis backend must be reachable and writable
|
||||
@item There must be a reachable JSON-RPC server at the other end of the signer socket path (see "signer" section in configurations)
|
||||
@end itemize
|
||||
|
||||
|
||||
@subsection tracker
|
||||
|
||||
Implements the @var{chainsyncer}, and registers the filters described in @strong{[ref transaction monitoring]} to be executed for every transaction. It consumes the appropriate @var{TASKS_TRANSFER_CALLBACKS} configuration setting to add externally defined filters at without having to change the daemon code.
|
||||
|
||||
The @var{tracker} has the same requisities for the @var{ContractRegistry} as the @var{tasker}.
|
||||
|
||||
@strong{Important! Guarantees of filter executions has some caveats. Refer to the @var{chainsyncer} documentation for more details.}
|
||||
|
||||
|
||||
@subsection dispatcher
|
||||
|
||||
Uses the @code{get_upcoming_tx} method call from @var{chainqueue} to receive batches of queued transactions that are ready to send to the blockchain node. Every batch will only contain a single transaction by any one address, which will be the transaction with the next nonce not previously seen by the network. There is no limit currently set to how many transactions that will be included in a single batch.
|
||||
|
||||
|
||||
@subsection retrier
|
||||
|
||||
The responsibility of the @var{retrier} is to re-queue transactions that failed to be sent to the blockchain node, as well as create @emph{replacements} for transactions whose processing by the network has been delayed. @strong{[refer transaction obolestion]}.
|
||||
|
||||
It is in turn the responsiblity of the @var{dispatcher} to send these (re-)queued transactions to the blockchain node.
|
||||
|
||||
|
||||
@section Tools
|
||||
|
||||
A collection of CLI tools have been provided to help with diagnostics and other administrative tasks. These use the same configuration infrastructure as the daemons.
|
||||
|
||||
Tool scripts are located in the @file{cic_eth.runnable} package. If @var{cic-eth} is installed as a python package, they are installed as executables in @var{PATH}.
|
||||
|
||||
|
||||
@subsection info (cic-eth-info)
|
||||
|
||||
Returns self-explanatory metadata for the blockchain network, and optionally an address.
|
||||
|
||||
|
||||
@subsection inspect (cic-eth-inspect)
|
||||
|
||||
Returns information about a specific resource related to the tranasaction queue. The results returned depend on the type of the argument.
|
||||
|
||||
@table @var
|
||||
@item registry_address
|
||||
Ethereum address of the @var{ContractRegistry} contract
|
||||
@item chain_spec
|
||||
String representation of the connected blockchain according to the @var{chainlib} @var{ChainSpec} format.
|
||||
@item tx_retry_delay
|
||||
Minimum time in seconds to wait before retrying a transaction
|
||||
@item trust_address
|
||||
Comma-separated list of one or more ethereum addresses regarded as trusted for describing other resources, Used by @var{cic-eth-registry} in the context of the @var{AddressDeclarator}.
|
||||
@item defalt_token_symbol
|
||||
Fallback token to operate on when no other context is given.
|
||||
@item health_modules
|
||||
Comma-separated list of methods to execute liveness tests against. (see ...)
|
||||
@item run_dir
|
||||
Directory to use for session-scoped variables for @var{cic-eth} daemon parent processes.
|
||||
@item lock
|
||||
If the argument is the literal string @kbd{lock}, it will list all active lock settings currently in effect. (see @strong{[reference locking section]})
|
||||
@item <address>
|
||||
If the argument is a 0x-prefixed hex string of 42 characters, it returns all transactions where the specified address is a sender or recipient@footnote{If the address is the gas gifter or the accounts index writer, this may be a @emph{lot} of transactions. Use with care!}
|
||||
@item <tx_hash>
|
||||
If the argument is a 0x-prefixed hex string of 66 characters, it returns data from the custodial queueing system aswell as the network for a single transaction whose hash matches the input. Fails if the transaction does not exist in the queue
|
||||
@item <code>
|
||||
If the argument is a 0x-prefixed hex string longer than 66 bytes, the argument will be interpreted as raw RLP serialized transaction data, and attempt to match this with an entry in the queue. If a match is found, the result is the same as for @var{<tx_hash>}
|
||||
@end table
|
||||
|
||||
|
||||
@subsection celery
|
||||
@subsection create (cic-eth-create)
|
||||
|
||||
@table @var
|
||||
@item broker_url
|
||||
Message broker URL
|
||||
@item result_url
|
||||
Result backend URL
|
||||
@item debug
|
||||
Boolean value. If set, the amount of available context for a task in the result backend will be maximized@footnote{THis is a @emph{required} setting for the task graph documenter to enabled it to display task names in the graph}.
|
||||
@end table
|
||||
Create a new account, optionally registering the account in the accounts registry, and optionally receiving the newly created address through a redis subscription.
|
||||
|
||||
@subsection transfer (cic-eth-transfer)
|
||||
|
||||
@subsection database
|
||||
Execute a token transfer on behalf of a custodial account.
|
||||
|
||||
See ref cic-base when ready
|
||||
@subsection tag (cic-eth-tag)
|
||||
|
||||
Associate an account address with a string identifier. See @strong{[reference SYSTEM ACCOUNTS]}.
|
||||
|
||||
@subsection eth
|
||||
@subsection ctrl (cic-eth-ctrl)
|
||||
|
||||
@table @var
|
||||
@item provider
|
||||
Address of default RPC endpoint for transactions and state queries.
|
||||
@item gas_gifter_minimum_balance
|
||||
The minimum gas balance that must be held by the @code{GAS GIFTER} token before the queue processing shuts down@footnote{You should really make sure that this threshold is never hit}
|
||||
@end table
|
||||
Set or reset lock bits, globally or per account address.
|
||||
|
||||
@subsection resend (cic-eth-resend)
|
||||
|
||||
@subsection redis
|
||||
|
||||
Defines connection to the redis server used outside of the context of @var{celery}. This is usually the same server, but should be a different db.
|
||||
|
||||
@table @var
|
||||
@item host
|
||||
Redis hostname
|
||||
@item port
|
||||
Redis port
|
||||
@item db
|
||||
Redis db
|
||||
@end table
|
||||
|
||||
|
||||
@subsection signer
|
||||
|
||||
Parameters
|
||||
|
||||
@table @var
|
||||
@item socket_path
|
||||
The connection string for the signer JSON-RPC service.@footnote{The @var{crypto-dev-signer} supports UNIX socket or a HTTP(S) connections}
|
||||
@item secret
|
||||
If set, this password is used to add obfuscation on top of the encryption already applied by the signer for the keystore.
|
||||
@end table
|
||||
|
||||
|
||||
|
||||
@subsection ssl
|
||||
|
||||
Certificate information for https api callbacks.
|
||||
|
||||
@table @var
|
||||
@item enable_client
|
||||
Boolean value. If set, client certificate will be used to authenticate the callback request.
|
||||
@item cert_file
|
||||
Client certificate file in PEM or DER format
|
||||
@item key_file
|
||||
Client key file in PEM or DER format
|
||||
@item password
|
||||
Password for unlocking the client key
|
||||
@item ca_file
|
||||
Certificate authority bundle, to verify the certificate sent by the callback server.
|
||||
@end table
|
||||
|
||||
|
||||
@subsection syncer
|
||||
|
||||
@table @var
|
||||
@item loop_interval
|
||||
Seconds to pause before each execution of the @var{chainsyncer} poll loop.
|
||||
@end table
|
||||
Resend a transaction. This can either be done "in-place," which means increasing the gas price and re-queueing@footnote{this is the same thing that the retrier does}. It can also be used to @emph{clone} a transaction, which obviously will duplicate the effect of the cloned transaction on the blockchain network.
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue