mirror of
https://github.com/GrassrootsEconomics/cic-dw.git
synced 2025-02-02 10:47:35 +01:00
docs: add description and README
This commit is contained in:
parent
8288880c79
commit
18eafe9694
13
README.md
Normal file
13
README.md
Normal file
@ -0,0 +1,13 @@
|
|||||||
|
![GitHub release (latest by date)](https://img.shields.io/github/v/release/grassrootseconomics/cic-dw)
|
||||||
|
![GitHub Workflow Status](https://img.shields.io/github/workflow/status/grassrootseconomics/cic-dw/goreleaser)
|
||||||
|
|
||||||
|
# cic-dw
|
||||||
|
|
||||||
|
> CIC Data Warehouse
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- [DB Schema](docs/cic-dw-schema.png)
|
||||||
|
- [Syncers](docs/syncers.md)
|
||||||
|
- [API](docs/api.md)
|
||||||
|
|
9
docs/api.md
Normal file
9
docs/api.md
Normal file
@ -0,0 +1,9 @@
|
|||||||
|
## API
|
||||||
|
|
||||||
|
The data warehouse additionally exposes a couple of REST API's (GraphQL planned):
|
||||||
|
|
||||||
|
1. Dashboard API (`/dashboard`) - Exposes data for [`cic-dashboard`](https://github.com/grassrootseconomics/cic-dashboard). Most data is expected to be chart/table API specific and usually not human readable.
|
||||||
|
2. Public API (`/public`) - Exposes public (on-chain only/non-sensitive) data.
|
||||||
|
3. Internal API (planned)
|
||||||
|
|
||||||
|
Each API is domain separated i.e. separate SQL query files and router control.
|
27
docs/syncers.md
Normal file
27
docs/syncers.md
Normal file
@ -0,0 +1,27 @@
|
|||||||
|
## Syncers
|
||||||
|
|
||||||
|
Syncers are responsible for pulling in data from multiple sources into this central data warehouse. There are primarily 3 types of syncers:
|
||||||
|
|
||||||
|
1. On-chain syncer - Pulls in data from smart contracts storages, events e.t.c.
|
||||||
|
2. DB syncer - Sync from other Postgres databases through [`postgres_fdw`](https://www.postgresql.org/docs/current/postgres-fdw.html) which may rely on other syncers on their end.
|
||||||
|
3. API syncer - Pull in data from an exposed endpoint.
|
||||||
|
|
||||||
|
Syncers are implemented as periodic jobs and rely on cursor based pagination across all types of syncers to efficiently update the data. In some cases where the data is regularly updated, we either listen for events (if the remote is capable of sending events) or spawn periodic jobs for every entry.
|
||||||
|
|
||||||
|
### Syncers implemented
|
||||||
|
|
||||||
|
#### Cache syncer
|
||||||
|
|
||||||
|
Relies on the `cic_cache` db which is backed by cic-cache-tracker. Pulls in all on-chain transfer transactions.
|
||||||
|
|
||||||
|
#### Ussd syncer
|
||||||
|
|
||||||
|
Relies on the `cic_ussd` db. Pulls in relevant ussd account registration details and doubles up as an alternative to the on-chain account registry index.
|
||||||
|
|
||||||
|
#### Token syncer
|
||||||
|
|
||||||
|
On-chain syncer. Pulls in all registered CIC tokens from the token index.
|
||||||
|
|
||||||
|
#### Meta syncer (WIP)
|
||||||
|
|
||||||
|
API syncer. Pulls in all relevant user metadata from the CRDT backed meta API.
|
Loading…
Reference in New Issue
Block a user