diff --git a/README.md b/README.md index 98fdac9..82f0727 100644 --- a/README.md +++ b/README.md @@ -6,15 +6,83 @@ > CIC Chain Events -Filters live (and past) transactions on Celo and emits relevant events to a sink for further processing/indexing. +Filters live (and past) transactions on Celo and emits relevant transfer events to a NATS JetStream sink for further processing/indexing. -## Documentation +## Prerequisites -- [Config and usage](docs/usage.md) -- [Functionality](docs/functionality.md) -- [Writing filters](docs/filters.md) -- [API](docs/api.md) +- Linux OS (amd64) or Docker +- Postgres >= 14 +- Celo geth with GraphQL API enabled +- NATS server with JetStream enabled + +## Usage + +The provided `docker-compose.yaml` is the fastest way to get up and running. Bring up the Postgres and NATS conatiners with `docker-compose up -d` + +### 1. Run migrations + +Run the SQL migrations inside the `migrations` folder with `psql` or [`tern`](https://github.com/jackc/tern) (recommended). + +### 2. Update the config + +The base config is described in `config.toml`. Values can be overriden with env variables e.g. to disable metrics, set `METRICS_GO_PROCESS=false`. + +### 3. Start the service + +#### Compiling binary + +Run `make build` or download pre-compiled binaries from the [releases](https://github.com/grassrootseconomics/cic-chain-events/releases) page. + +Then start the service with `./cic-chain-events` + +Optional flags: + +- `-config` - `config.toml` file path +- `-debug` - Enable/disable debug level logs +- `-queries` - `queries.sql` file path + +#### Docker + +To pull the pre-built docker image: + +`docker pull ghcr.io/grassrootseconomics/cic-chain-events/cic-chain-events:latest` + +Or to build it: + +`DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1 docker-compose -f docker-compose.build.yaml build --progress plain` + +### 4. NATS JetStream consumer + +A consumer with the following NATS JetStream config is required: + +- Durable +- Stream: `CHAIN.*` (See `config.toml` for stream subjects) + +[Benthos](https://benthos.dev) (Benthos can act as a JetStream consumer) example. + +```yaml +# config.yaml +input: + label: jetstream + nats_jetstream: + urls: + - nats://127.0.0.1:4222 + subject: "CHAIN.*" + durable: benthos + deliver: all +output: + stdout: + codec: lines +``` + +Then run: + +`benthos -c config.yaml` + +## Functionality + +[Read more here](docs/functionality.md). ## License -[AGPL-3.0](LICENSE) +[AGPL-3.0](LICENSE). diff --git a/docs/api.md b/docs/api.md deleted file mode 100644 index 828e482..0000000 --- a/docs/api.md +++ /dev/null @@ -1,4 +0,0 @@ -## API - -- `/stats` - Syncer stats (Ready after 1st successful janitor sweep). -- `/metrics` - Go process metrics (Prometheus format). \ No newline at end of file diff --git a/docs/filters.md b/docs/filters.md deleted file mode 100644 index 1a4512b..0000000 --- a/docs/filters.md +++ /dev/null @@ -1,11 +0,0 @@ -## Writing filters - -Filters must conform to the interface: - -```go -type Filter interface { - Execute(ctx context.Context, inputTransaction fetch.Transaction) (next bool, err error) -} -``` - -See examples in the `internal/filter` folder. diff --git a/docs/functionality.md b/docs/functionality.md index c2ab85f..d01b746 100644 --- a/docs/functionality.md +++ b/docs/functionality.md @@ -1,52 +1,38 @@ ## Functionality +## Filters + +Filters are initialized in `cmd/filters.go` and implemented in `internal/filters/*.go` folder. You will need to modify these files to suite your indexing needs. + +The existing implementation demo's tracking Celo stables transfer events and gives a rough idea on how to write filters. The final filter should always emit an event to NATS JetStream. + +## Syncers + ### Head syncer -Opens a websocket connection and processes live transactions. +The head syncer processes newely produced blocks independently by connection to the geth websocket endpoint. ### Janitor -Periodically checks for missed (and historical) blocks missed by the head syncer and queues them for processing. A gap range is processed twice to guarantee there is no missing block. +The janitor syncer checks for missing (blocks) gaps in the commited block sequence and queues them for processing. It can also function as a historical syncer too process older blocks. -### Pipeline +With the default `config.toml`, The janitor can process around 950-1000 blocks/min. -Fetches a block and executes the filters in serial order for every transaction in the block before finally committing the block to the store. +_Ordering_ -### Filter +Missed/historical blocks are not guaranteed to be processed in order, however a low concurrency setting would somewhat give an "in-order" behaviour (not to be relied upon in any case). -Processes a transaction and passes it on to the next filter or terminates the pipeline for that transaction if it is irrelevant. +## Block fetchers -### Store schema +The default GraphQL block fetcher is the recommended fetcher. An experimental RPC fetcher implementation is also provided as an example. + +## Pipeline + +The pipeline fetches a whole block with its full transaction and receipt objects, executes all loaded filters serially and finally commits the block value to the db. Blocks are processed atomically by the pipeline; a failure in one of the filters will trigger the janitor to re-queue the block and process the block again. + +## Store + +The postgres store keeps track of commited blocks and syncer curosors. Schema: - The `blocks` table keeps track of processed blocks. - The `syncer_meta` table keeps track of the lower_bound cursor. Below the lower_bound cursor, all blocks are guarnteed to have been processsed hence it is safe to trim the `blocks` table below that pointer. - -### GraphQL - -- Fetches a block (and some of its header details), transactions and transaction receipts embedded within the transaction object in a single call. - -### NATS JetStream - -- The final filter will emit an event to JetStream. - -To view/debug the JetStream messages, you can use [Benthos](https://benthos.dev) - -With a config like: - -```yaml -input: - label: jetstream - nats_jetstream: - urls: - - nats://127.0.0.1:4222 - subject: "CHAIN.*" - durable: benthos - deliver: all -output: - stdout: - codec: lines -``` - -## Caveats - -- Blocks are not guaranteed to be processed in order, however a low concurrency setting would somewhat give an "in-order" behaviour (not to be relied upon in any case). diff --git a/docs/usage.md b/docs/usage.md deleted file mode 100644 index f5d01cb..0000000 --- a/docs/usage.md +++ /dev/null @@ -1,36 +0,0 @@ -## Requirements - -- Celo (geth) node with GraphQL enabled -- Postgres 14+ - -## Running - -### 1. Run migrations - -Run the migrations inside the `migrations` folder. - -### 2. Update the config - -The base config is described in `config.toml`. Values can be overriden with env variables e.g. to disable metrics, set `METRICS_GO_PROCESS=false`. - -### 3. Start the service: - -**Compiling**: - -- Requires CGO_ENABLED=1 -- Prebuilt binaries (for amd64 only) available on the releases page - -**Docker**: - -- `docker pull ghcr.io/grassrootseconomics/cic-chain-events/cic-chain-events:latest` - -After compiling or within a Docker container: - -`$ ./cic-chain-events` - -Optional flags: - -- `-config` - `config.toml` file path -- `-debug` - Enable/disable debug level logs -- `-queries` - `queries.sql` file path -