docs: improve instructions and description

README.md

@@ -6,15 +6,83 @@
 
 > CIC Chain Events
 
-Filters live (and past) transactions on Celo and emits relevant events to a sink for further processing/indexing.
+Filters live (and past) transactions on Celo and emits relevant transfer events to a NATS JetStream sink for further processing/indexing.
 
-## Documentation
+## Prerequisites
 
-- [Config and usage](docs/usage.md)
+- Linux OS (amd64) or Docker
-- [Functionality](docs/functionality.md)
+- Postgres >= 14
-- [Writing filters](docs/filters.md)
+- Celo geth with GraphQL API enabled
-- [API](docs/api.md)
+- NATS server with JetStream enabled
+
+## Usage
+
+The provided `docker-compose.yaml` is the fastest way to get up and running. Bring up the Postgres and NATS conatiners with `docker-compose up -d`
+
+### 1. Run migrations
+
+Run the SQL migrations inside the `migrations` folder with `psql` or [`tern`](https://github.com/jackc/tern) (recommended).
+
+### 2. Update the config
+
+The base config is described in `config.toml`. Values can be overriden with env variables e.g. to disable metrics, set `METRICS_GO_PROCESS=false`.
+
+### 3. Start the service
+
+#### Compiling binary
+
+Run `make build` or download pre-compiled binaries from the [releases](https://github.com/grassrootseconomics/cic-chain-events/releases) page.
+
+Then start the service with `./cic-chain-events`
+
+Optional flags:
+
+- `-config` - `config.toml` file path
+- `-debug` - Enable/disable debug level logs
+- `-queries` - `queries.sql` file path
+
+#### Docker
+
+To pull the pre-built docker image:
+
+`docker pull ghcr.io/grassrootseconomics/cic-chain-events/cic-chain-events:latest`
+
+Or to build it:
+
+`DOCKER_BUILDKIT=1 COMPOSE_DOCKER_CLI_BUILD=1 docker-compose -f docker-compose.build.yaml build --progress plain`
+
+### 4. NATS JetStream consumer
+
+A consumer with the following NATS JetStream config is required:
+
+- Durable
+- Stream: `CHAIN.*` (See `config.toml` for stream subjects)
+
+[Benthos](https://benthos.dev) (Benthos can act as a JetStream consumer) example.
+
+```yaml
+# config.yaml
+input:
+  label: jetstream
+  nats_jetstream:
+    urls:
+      - nats://127.0.0.1:4222
+    subject: "CHAIN.*"
+    durable: benthos
+    deliver: all
+output:
+  stdout:
+    codec: lines
+```
+
+Then run:
+
+`benthos -c config.yaml`
+
+## Functionality
+
+[Read more here](docs/functionality.md).
 
 ## License
 
-[AGPL-3.0](LICENSE)
+[AGPL-3.0](LICENSE).

docs/api.md

@@ -1,4 +0,0 @@
-## API
-
-- `/stats`   - Syncer stats (Ready after 1st successful janitor sweep).
-- `/metrics` - Go process metrics (Prometheus format).

docs/filters.md

@@ -1,11 +0,0 @@
-## Writing filters
-
-Filters must conform to the interface:
-
-```go
-type Filter interface {
-	Execute(ctx context.Context, inputTransaction fetch.Transaction) (next bool, err error)
-}
-```
-
-See examples in the `internal/filter` folder.

docs/functionality.md

@@ -1,52 +1,38 @@
 ## Functionality
 
+## Filters
+
+Filters are initialized in `cmd/filters.go` and implemented in `internal/filters/*.go` folder. You will need to modify these files to suite your indexing needs.
+
+The existing implementation demo's tracking Celo stables transfer events and gives a rough idea on how to write filters. The final filter should always emit an event to NATS JetStream.
+
+## Syncers
+
 ### Head syncer
 
-Opens a websocket connection and processes live transactions.
+The head syncer processes newely produced blocks independently by connection to the geth websocket endpoint.
 
 ### Janitor
 
-Periodically checks for missed (and historical) blocks missed by the head syncer and queues them for processing. A gap range is processed twice to guarantee there is no missing block.
+The janitor syncer checks for missing (blocks) gaps in the commited block sequence and queues them for processing. It can also function as a historical syncer too process older blocks.
 
-### Pipeline
+With the default `config.toml`, The janitor can process around 950-1000 blocks/min.
 
-Fetches a block and executes the filters in serial order for every transaction in the block before finally committing the block to the store.
+_Ordering_
 
-### Filter
+Missed/historical blocks are not guaranteed to be processed in order, however a low concurrency setting would somewhat give an "in-order" behaviour (not to be relied upon in any case).
 
-Processes a transaction and passes it on to the next filter or terminates the pipeline for that transaction if it is irrelevant.
+## Block fetchers
 
-### Store schema
+The default GraphQL block fetcher is the recommended fetcher. An experimental RPC fetcher implementation is also provided as an example.
+
+## Pipeline
+
+The pipeline fetches a whole block with its full transaction and receipt objects, executes all loaded filters serially and finally commits the block value to the db. Blocks are processed atomically by the pipeline; a failure in one of the filters will trigger the janitor to re-queue the block and process the block again.
+
+## Store
+
+The postgres store keeps track of commited blocks and syncer curosors. Schema:
 
 - The `blocks` table keeps track of processed blocks.
 - The `syncer_meta` table keeps track of the lower_bound cursor. Below the lower_bound cursor, all blocks are guarnteed to have been processsed hence it is safe to trim the `blocks` table below that pointer.
-
-### GraphQL
-
-- Fetches a block (and some of its header details), transactions and transaction receipts embedded within the transaction object in a single call.
-
-### NATS JetStream
-
-- The final filter will emit an event to JetStream.
-
-To view/debug the JetStream messages, you can use [Benthos](https://benthos.dev)
-
-With a config like:
-
-```yaml
-input:
-  label: jetstream
-  nats_jetstream:
-    urls:
-      - nats://127.0.0.1:4222
-    subject: "CHAIN.*"
-    durable: benthos
-    deliver: all
-output:
-  stdout:
-    codec: lines
-```
-
-## Caveats
-
-- Blocks are not guaranteed to be processed in order, however a low concurrency setting would somewhat give an "in-order" behaviour (not to be relied upon in any case).

docs/usage.md

@@ -1,36 +0,0 @@
-## Requirements
-
-- Celo (geth) node with GraphQL enabled
-- Postgres 14+
-
-## Running
-
-### 1. Run migrations
-
-Run the migrations inside the `migrations` folder.
-
-### 2. Update the config
-
-The base config is described in `config.toml`. Values can be overriden with env variables e.g. to disable metrics, set `METRICS_GO_PROCESS=false`.
-
-### 3. Start the service:
-
-**Compiling**:
-
-- Requires CGO_ENABLED=1
-- Prebuilt binaries (for amd64 only) available on the releases page
-
-**Docker**:
-
-- `docker pull ghcr.io/grassrootseconomics/cic-chain-events/cic-chain-events:latest`
-
-After compiling or within a Docker container:
-
-`$ ./cic-chain-events`
-
-Optional flags:
-
-- `-config` - `config.toml` file path
-- `-debug` - Enable/disable debug level logs
-- `-queries` - `queries.sql` file path
-