cic-chain-events/docs/functionality.md

## Functionality

## Filters

Filters are initialized in `cmd/filters.go` and implemented in `internal/filters/*.go` folder. You will need to modify these files to suite your indexing needs.

The existing implementation demo's tracking Celo stables transfer events and gives a rough idea on how to write filters. The final filter should always emit an event to NATS JetStream.

## Syncers

### Head syncer

The head syncer processes newely produced blocks independently by connecting to the geth websocket endpoint.

### Janitor

The janitor syncer checks for missing (blocks) gaps in the commited block sequence and queues them for processing. It can also function as a historical syncer to process older blocks.

With the default `config.toml`, The janitor can process around 950-1000 blocks/min.

**Ordering**

Missed/historical blocks are not guaranteed to be processed in order, however a low concurrency setting would somewhat give an "in-order" behaviour (not to be relied upon in any case).

## Block fetchers

The default GraphQL block fetcher is the recommended fetcher. An experimental RPC fetcher implementation is also provided as an example.

## Pipeline

The pipeline fetches a whole block with its full transaction and receipt objects, executes all loaded filters serially and finally commits the block number to the db. Blocks are processed atomically by the pipeline; a failure in one of the filters will trigger the janitor to re-queue the block and process the block again.

## Store

The postgres store keeps track of commited blocks and syncer curosors. Schema:

- The `blocks` table keeps track of processed blocks.
- The `syncer_meta` table keeps track of the lower_bound cursor. Below the lower_bound cursor, all blocks are guarnteed to have been processsed hence it is safe to trim the `blocks` table below that pointer.