diff --git a/README.md b/README.md new file mode 100644 index 0000000..4f7edd6 --- /dev/null +++ b/README.md @@ -0,0 +1,13 @@ +![GitHub release (latest by date)](https://img.shields.io/github/v/release/grassrootseconomics/cic-dw) +![GitHub Workflow Status](https://img.shields.io/github/workflow/status/grassrootseconomics/cic-dw/goreleaser) + +# cic-dw + +> CIC Data Warehouse + +## Documentation + +- [DB Schema](docs/cic-dw-schema.png) +- [Syncers](docs/syncers.md) +- [API](docs/api.md) + diff --git a/docs/api.md b/docs/api.md new file mode 100644 index 0000000..cf489d4 --- /dev/null +++ b/docs/api.md @@ -0,0 +1,9 @@ +## API + +The data warehouse additionally exposes a couple of REST API's (GraphQL planned): + +1. Dashboard API (`/dashboard`) - Exposes data for [`cic-dashboard`](https://github.com/grassrootseconomics/cic-dashboard). Most data is expected to be chart/table API specific and usually not human readable. +2. Public API (`/public`) - Exposes public (on-chain only/non-sensitive) data. +3. Internal API (planned) + +Each API is domain separated i.e. separate SQL query files and router control. \ No newline at end of file diff --git a/docs/syncers.md b/docs/syncers.md new file mode 100644 index 0000000..98444bc --- /dev/null +++ b/docs/syncers.md @@ -0,0 +1,27 @@ +## Syncers + +Syncers are responsible for pulling in data from multiple sources into this central data warehouse. There are primarily 3 types of syncers: + +1. On-chain syncer - Pulls in data from smart contracts storages, events e.t.c. +2. DB syncer - Sync from other Postgres databases through [`postgres_fdw`](https://www.postgresql.org/docs/current/postgres-fdw.html) which may rely on other syncers on their end. +3. API syncer - Pull in data from an exposed endpoint. + +Syncers are implemented as periodic jobs and rely on cursor based pagination across all types of syncers to efficiently update the data. In some cases where the data is regularly updated, we either listen for events (if the remote is capable of sending events) or spawn periodic jobs for every entry. + +### Syncers implemented + +#### Cache syncer + +Relies on the `cic_cache` db which is backed by cic-cache-tracker. Pulls in all on-chain transfer transactions. + +#### Ussd syncer + +Relies on the `cic_ussd` db. Pulls in relevant ussd account registration details and doubles up as an alternative to the on-chain account registry index. + +#### Token syncer + +On-chain syncer. Pulls in all registered CIC tokens from the token index. + +#### Meta syncer (WIP) + +API syncer. Pulls in all relevant user metadata from the CRDT backed meta API. \ No newline at end of file