docs: improve instructions and description
This commit is contained in:
parent
2161e8936b
commit
7923384328
GPG Key ID:
7DD45520C01CD85D
82
README.md
82
README.md
|
|
@ -6,15 +6,83 @@
|
|||
|
||||
> CIC Chain Events
|
||||
|
||||
Filters live (and past) transactions on Celo and emits relevant events to a sink for further processing/indexing.
|
||||
Filters live (and past) transactions on Celo and emits relevant transfer events to a NATS JetStream sink for further processing/indexing.
|
||||
|
||||
## Documentation
|
||||
## Prerequisites
|
||||
|
||||
- [Config and usage](docs/usage.md)
|
||||
- [Functionality](docs/functionality.md)
|
||||
- [Writing filters](docs/filters.md)
|
||||
- [API](docs/api.md)
|
||||
- Linux OS (amd64) or Docker
|
||||
- Postgres >= 14
|
||||
- Celo geth with GraphQL API enabled
|
||||
- NATS server with JetStream enabled
|
||||
|
||||
## Usage
|
||||
|
||||
The provided `docker-compose.yaml` is the fastest way to get up and running. Bring up the Postgres and NATS conatiners with `docker-compose up -d`
|
||||
|
||||
### 1. Run migrations
|
||||
|
||||
Run the SQL migrations inside the `migrations` folder with `psql` or [`tern`](https://github.com/jackc/tern) (recommended).
|
||||
|
||||
### 2. Update the config
|
||||
|
||||
The base config is described in `config.toml`. Values can be overriden with env variables e.g. to disable metrics, set `METRICS_GO_PROCESS=false`.
|
||||
|
||||
### 3. Start the service
|
||||
|
||||
#### Compiling binary
|
||||
|
||||
Run `make build` or download pre-compiled binaries from the [releases](https://github.com/grassrootseconomics/cic-chain-events/releases) page.
|
||||
|
||||
Then start the service with `./cic-chain-events`
|
||||
|
||||
Optional flags:
|
||||
|
||||
- `-config` - `config.toml` file path
|
||||
- `-debug` - Enable/disable debug level logs
|
||||
- `-queries` - `queries.sql` file path
|
||||
|
||||
#### Docker
|
||||
|
||||
To pull the pre-built docker image:
|
||||
|
||||
`docker pull ghcr.io/grassrootseconomics/cic-chain-events/cic-chain-events:latest`
|
||||
|
||||
Or to build it:
|
||||
|
||||
`DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1 docker-compose -f docker-compose.build.yaml build --progress plain`
|
||||
|
||||
### 4. NATS JetStream consumer
|
||||
|
||||
A consumer with the following NATS JetStream config is required:
|
||||
|
||||
- Durable
|
||||
- Stream: `CHAIN.*` (See `config.toml` for stream subjects)
|
||||
|
||||
[Benthos](https://benthos.dev) (Benthos can act as a JetStream consumer) example.
|
||||
|
||||
```yaml
|
||||
# config.yaml
|
||||
input:
|
||||
label: jetstream
|
||||
nats_jetstream:
|
||||
urls:
|
||||
- nats://127.0.0.1:4222
|
||||
subject: "CHAIN.*"
|
||||
durable: benthos
|
||||
deliver: all
|
||||
output:
|
||||
stdout:
|
||||
codec: lines
|
||||
```
|
||||
|
||||
Then run:
|
||||
|
||||
`benthos -c config.yaml`
|
||||
|
||||
## Functionality
|
||||
|
||||
[Read more here](docs/functionality.md).
|
||||
|
||||
## License
|
||||
|
||||
[AGPL-3.0](LICENSE)
|
||||
[AGPL-3.0](LICENSE).
|
||||
|
|
|
|||
|
|
@ -1,4 +0,0 @@
|
|||
## API
|
||||
|
||||
- `/stats` - Syncer stats (Ready after 1st successful janitor sweep).
|
||||
- `/metrics` - Go process metrics (Prometheus format).
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
## Writing filters
|
||||
|
||||
Filters must conform to the interface:
|
||||
|
||||
```go
|
||||
type Filter interface {
|
||||
Execute(ctx context.Context, inputTransaction fetch.Transaction) (next bool, err error)
|
||||
}
|
||||
```
|
||||
|
||||
See examples in the `internal/filter` folder.
|
||||
|
|
@ -1,52 +1,38 @@
|
|||
## Functionality
|
||||
|
||||
## Filters
|
||||
|
||||
Filters are initialized in `cmd/filters.go` and implemented in `internal/filters/*.go` folder. You will need to modify these files to suite your indexing needs.
|
||||
|
||||
The existing implementation demo's tracking Celo stables transfer events and gives a rough idea on how to write filters. The final filter should always emit an event to NATS JetStream.
|
||||
|
||||
## Syncers
|
||||
|
||||
### Head syncer
|
||||
|
||||
Opens a websocket connection and processes live transactions.
|
||||
The head syncer processes newely produced blocks independently by connection to the geth websocket endpoint.
|
||||
|
||||
### Janitor
|
||||
|
||||
Periodically checks for missed (and historical) blocks missed by the head syncer and queues them for processing. A gap range is processed twice to guarantee there is no missing block.
|
||||
The janitor syncer checks for missing (blocks) gaps in the commited block sequence and queues them for processing. It can also function as a historical syncer too process older blocks.
|
||||
|
||||
### Pipeline
|
||||
With the default `config.toml`, The janitor can process around 950-1000 blocks/min.
|
||||
|
||||
Fetches a block and executes the filters in serial order for every transaction in the block before finally committing the block to the store.
|
||||
_Ordering_
|
||||
|
||||
### Filter
|
||||
Missed/historical blocks are not guaranteed to be processed in order, however a low concurrency setting would somewhat give an "in-order" behaviour (not to be relied upon in any case).
|
||||
|
||||
Processes a transaction and passes it on to the next filter or terminates the pipeline for that transaction if it is irrelevant.
|
||||
## Block fetchers
|
||||
|
||||
### Store schema
|
||||
The default GraphQL block fetcher is the recommended fetcher. An experimental RPC fetcher implementation is also provided as an example.
|
||||
|
||||
## Pipeline
|
||||
|
||||
The pipeline fetches a whole block with its full transaction and receipt objects, executes all loaded filters serially and finally commits the block value to the db. Blocks are processed atomically by the pipeline; a failure in one of the filters will trigger the janitor to re-queue the block and process the block again.
|
||||
|
||||
## Store
|
||||
|
||||
The postgres store keeps track of commited blocks and syncer curosors. Schema:
|
||||
|
||||
- The `blocks` table keeps track of processed blocks.
|
||||
- The `syncer_meta` table keeps track of the lower_bound cursor. Below the lower_bound cursor, all blocks are guarnteed to have been processsed hence it is safe to trim the `blocks` table below that pointer.
|
||||
|
||||
### GraphQL
|
||||
|
||||
- Fetches a block (and some of its header details), transactions and transaction receipts embedded within the transaction object in a single call.
|
||||
|
||||
### NATS JetStream
|
||||
|
||||
- The final filter will emit an event to JetStream.
|
||||
|
||||
To view/debug the JetStream messages, you can use [Benthos](https://benthos.dev)
|
||||
|
||||
With a config like:
|
||||
|
||||
```yaml
|
||||
input:
|
||||
label: jetstream
|
||||
nats_jetstream:
|
||||
urls:
|
||||
- nats://127.0.0.1:4222
|
||||
subject: "CHAIN.*"
|
||||
durable: benthos
|
||||
deliver: all
|
||||
output:
|
||||
stdout:
|
||||
codec: lines
|
||||
```
|
||||
|
||||
## Caveats
|
||||
|
||||
- Blocks are not guaranteed to be processed in order, however a low concurrency setting would somewhat give an "in-order" behaviour (not to be relied upon in any case).
|
||||
|
|
|
|||
|
|
@ -1,36 +0,0 @@
|
|||
## Requirements
|
||||
|
||||
- Celo (geth) node with GraphQL enabled
|
||||
- Postgres 14+
|
||||
|
||||
## Running
|
||||
|
||||
### 1. Run migrations
|
||||
|
||||
Run the migrations inside the `migrations` folder.
|
||||
|
||||
### 2. Update the config
|
||||
|
||||
The base config is described in `config.toml`. Values can be overriden with env variables e.g. to disable metrics, set `METRICS_GO_PROCESS=false`.
|
||||
|
||||
### 3. Start the service:
|
||||
|
||||
**Compiling**:
|
||||
|
||||
- Requires CGO_ENABLED=1
|
||||
- Prebuilt binaries (for amd64 only) available on the releases page
|
||||
|
||||
**Docker**:
|
||||
|
||||
- `docker pull ghcr.io/grassrootseconomics/cic-chain-events/cic-chain-events:latest`
|
||||
|
||||
After compiling or within a Docker container:
|
||||
|
||||
`$ ./cic-chain-events`
|
||||
|
||||
Optional flags:
|
||||
|
||||
- `-config` - `config.toml` file path
|
||||
- `-debug` - Enable/disable debug level logs
|
||||
- `-queries` - `queries.sql` file path
|
||||
|
||||
Loading…
Reference in New Issue