docs: add description and README

This commit is contained in:
Mohamed Sohail 2022-05-14 11:06:42 +03:00
parent 8288880c79
commit 18eafe9694
Signed by: kamikazechaser
GPG Key ID: 7DD45520C01CD85D
3 changed files with 49 additions and 0 deletions

13
README.md Normal file
View File

@ -0,0 +1,13 @@
![GitHub release (latest by date)](https://img.shields.io/github/v/release/grassrootseconomics/cic-dw)
![GitHub Workflow Status](https://img.shields.io/github/workflow/status/grassrootseconomics/cic-dw/goreleaser)
# cic-dw
> CIC Data Warehouse
## Documentation
- [DB Schema](docs/cic-dw-schema.png)
- [Syncers](docs/syncers.md)
- [API](docs/api.md)

9
docs/api.md Normal file
View File

@ -0,0 +1,9 @@
## API
The data warehouse additionally exposes a couple of REST API's (GraphQL planned):
1. Dashboard API (`/dashboard`) - Exposes data for [`cic-dashboard`](https://github.com/grassrootseconomics/cic-dashboard). Most data is expected to be chart/table API specific and usually not human readable.
2. Public API (`/public`) - Exposes public (on-chain only/non-sensitive) data.
3. Internal API (planned)
Each API is domain separated i.e. separate SQL query files and router control.

27
docs/syncers.md Normal file
View File

@ -0,0 +1,27 @@
## Syncers
Syncers are responsible for pulling in data from multiple sources into this central data warehouse. There are primarily 3 types of syncers:
1. On-chain syncer - Pulls in data from smart contracts storages, events e.t.c.
2. DB syncer - Sync from other Postgres databases through [`postgres_fdw`](https://www.postgresql.org/docs/current/postgres-fdw.html) which may rely on other syncers on their end.
3. API syncer - Pull in data from an exposed endpoint.
Syncers are implemented as periodic jobs and rely on cursor based pagination across all types of syncers to efficiently update the data. In some cases where the data is regularly updated, we either listen for events (if the remote is capable of sending events) or spawn periodic jobs for every entry.
### Syncers implemented
#### Cache syncer
Relies on the `cic_cache` db which is backed by cic-cache-tracker. Pulls in all on-chain transfer transactions.
#### Ussd syncer
Relies on the `cic_ussd` db. Pulls in relevant ussd account registration details and doubles up as an alternative to the on-chain account registry index.
#### Token syncer
On-chain syncer. Pulls in all registered CIC tokens from the token index.
#### Meta syncer (WIP)
API syncer. Pulls in all relevant user metadata from the CRDT backed meta API.