update readme

README.md

@@ -1,42 +1,159 @@
 # cic-internal-integration
 
-## Getting started 
+## Backend Requirements
 
-## Make some keys
+* [Docker](https://www.docker.com/).
+* [Docker Compose](https://docs.docker.com/compose/install/).
 
-```
-docker build -t bloxie . && docker run -v "$(pwd)/keys:/root/keys" --rm -it -t bloxie account new --chain /root/bloxberg.json --keys-path /root/keys
+## Backend local development
+
+* Start the stack with Docker Compose:
+
+```bash
+docker-compose up -d
 ```
 
+* Now you can open your browser and interact with these URLs:
 
-### Prepare the repo
+Frontend (CICADA), built with Docker, with routes handled based on the path: http://localhost
 
-This is stuff we need to put in  makefile but for now...
+PGAdmin, PostgreSQL web administration: http://localhost:5050
 
-File mounts and permisssions need to be set
-```
-chmod -R 755 scripts/initdb apps/cic-meta/scripts/initdb
-````
+Flower, administration of Celery tasks: http://localhost:5555
 
-start cluster
-```
-docker-compose up
+Traefik UI, to see how the routes are being handled by the proxy: http://localhost:8090
+
+**Note**: The first time you start your stack, it might take a minute for it to be ready. While the backend waits for the database to be ready and configures everything. You can check the logs to monitor it.
+
+To check the logs, run:
+
+```bash
+docker-compose logs
 ```
 
-stop cluster
-```
-docker-compose down
+To check the logs of a specific service, add the name of the service, e.g.:
+
+```bash
+docker-compose logs backend
 ```
 
-delete data
-```
-docker-compose down -v
+If your Docker is not running in `localhost` (the URLs above wouldn't work) check the sections below on **Development with Docker Toolbox** and **Development with a custom IP**.
+
+## Backend local development, additional details
+
+**fill  me in**
+
+### Docker Compose Override
+
+During development, you can change Docker Compose settings that will only affect the local development environment, in the file `docker-compose.override.yml`.
+
+The changes to that file only affect the local development environment, not the production environment. So, you can add "temporary" changes that help the development workflow.
+
+For example, the directory with the backend code is mounted as a Docker "host volume", mapping the code you change live to the directory inside the container. That allows you to test your changes right away, without having to build the Docker image again. It should only be done during development, for production, you should build the Docker image with a recent version of the backend code. But during development, it allows you to iterate very fast.
+
+There is also a command override that runs `/start-reload.sh` (included in the base image) instead of the default `/start.sh` (also included in the base image). It starts a single server process (instead of multiple, as would be for production) and reloads the process whenever the code changes. Have in mind that if you have a syntax error and save the Python file, it will break and exit, and the container will stop. After that, you can restart the container by fixing the error and running again:
+
+```console
+$ docker-compose up -d
 ```
 
-rebuild an images
-```
-docker-compose up --build <service_name>
+There is also a commented out `command` override, you can uncomment it and comment the default one. It makes the backend container run a process that does "nothing", but keeps the container alive. That allows you to get inside your running container and execute commands inside, for example a Python interpreter to test installed dependencies, or start the development server that reloads when it detects changes, or start a Jupyter Notebook session.
+
+To get inside the container with a `bash` session you can start the stack with:
+
+```console
+$ docker-compose up -d
 ```
 
-Deployment variables are writtend to service-configs/.env after everthing is up.
+and then `exec` inside the running container:
+
+```console
+$ docker-compose exec backend bash
+```
+
+You should see an output like:
+
+```console
+root@7f2607af31c3:/app#
+```
+
+that means that you are in a `bash` session inside your container, as a `root` user, under the `/app` directory.
+
+There you can use the script `/start-reload.sh` to run the debug live reloading server. You can run that script from inside the container with:
+
+```console
+$ bash /start-reload.sh
+```
+
+...it will look like:
+
+```console
+root@7f2607af31c3:/app# bash /start-reload.sh
+```
+
+and then hit enter. That runs the live reloading server that auto reloads when it detects code changes.
+
+Nevertheless, if it doesn't detect a change but a syntax error, it will just stop with an error. But as the container is still alive and you are in a Bash session, you can quickly restart it after fixing the error, running the same command ("up arrow" and "Enter").
+
+...this previous detail is what makes it useful to have the container alive doing nothing and then, in a Bash session, make it run the live reload server.
+
+### Backend tests
+
+To test the backend run:
+
+```console
+$ DOMAIN=backend sh ./scripts/test.sh
+```
+
+The file `./scripts/test.sh` has the commands to generate a testing `docker-stack.yml` file, start the stack and test it.
+
+The tests run with Pytest, modify and add tests to `./backend/app/app/tests/`.
+
+If you use GitLab CI the tests will run automatically.
+
+#### Local tests
+
+Start the stack with this command:
+
+```Bash
+DOMAIN=backend sh ./scripts/test-local.sh
+```
+The `./backend/app` directory is mounted as a "host volume" inside the docker container (set in the file `docker-compose.dev.volumes.yml`).
+You can rerun the test on live code:
+
+```Bash
+docker-compose exec backend /app/tests-start.sh
+```
+
+#### Test running stack
+
+If your stack is already up and you just want to run the tests, you can use:
+
+```bash
+docker-compose exec backend /app/tests-start.sh
+```
+
+That `/app/tests-start.sh` script just calls `pytest` after making sure that the rest of the stack is running. If you need to pass extra arguments to `pytest`, you can pass them to that command and they will be forwarded.
+
+For example, to stop on first error:
+
+```bash
+docker-compose exec backend bash /app/tests-start.sh -x
+```
+
+#### Test Coverage
+
+Because the test scripts forward arguments to `pytest`, you can enable test coverage HTML report generation by passing `--cov-report=html`.
+
+To run the local tests with coverage HTML reports:
+
+```Bash
+DOMAIN=backend sh ./scripts/test-local.sh --cov-report=html
+```
+
+To run the tests in a running stack with coverage HTML reports:
+
+```bash
+docker-compose exec backend bash /app/tests-start.sh --cov-report=html
+```
 

apps/cic-staff-client/src/assets/js/ethtx/hex.d.ts

@@ -0,0 +1,5 @@
+declare function strip0x(hexString: string): string;
+declare function add0x(hexString: string): string;
+declare function fromHex(hexString: string): Uint8Array;
+declare function toHex(bytes: Uint8Array): string;
+export { fromHex, toHex, strip0x, add0x, };

apps/cic-staff-client/src/assets/js/ethtx/hex.js

@@ -0,0 +1,41 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.add0x = exports.strip0x = exports.toHex = exports.fromHex = void 0;
+// improve
+function validHex(hexString) {
+    return hexString;
+}
+function even(hexString) {
+    if (hexString.length % 2 != 0) {
+        hexString = '0' + hexString;
+    }
+    return hexString;
+}
+function strip0x(hexString) {
+    if (hexString.length < 2) {
+        throw new Error('invalid hex');
+    }
+    else if (hexString.substring(0, 2) == '0x') {
+        hexString = hexString.substring(2);
+    }
+    return validHex(even(hexString));
+}
+exports.strip0x = strip0x;
+function add0x(hexString) {
+    if (hexString.length < 2) {
+        throw new Error('invalid hex');
+    }
+    else if (hexString.substring(0, 2) != '0x') {
+        hexString = '0x' + hexString;
+    }
+    return validHex(even(hexString));
+}
+exports.add0x = add0x;
+function fromHex(hexString) {
+    return new Uint8Array(hexString.match(/.{1,2}/g).map(function (byte) { return parseInt(byte, 16); }));
+}
+exports.fromHex = fromHex;
+function toHex(bytes) {
+    return bytes.reduce(function (str, byte) { return str + byte.toString(16).padStart(2, '0'); }, '');
+}
+exports.toHex = toHex;

apps/cic-staff-client/src/assets/js/ethtx/index.d.ts

@@ -0,0 +1 @@
+export { Tx } from './tx';

apps/cic-staff-client/src/assets/js/ethtx/index.js

@@ -0,0 +1,5 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.Tx = void 0;
+var tx_1 = require("./tx");
+Object.defineProperty(exports, "Tx", { enumerable: true, get: function () { return tx_1.Tx; } });

apps/cic-staff-client/src/assets/js/ethtx/tx.d.ts

@@ -0,0 +1,29 @@
+declare function toValue(n: number): bigint;
+declare function stringToValue(s: string): bigint;
+declare function hexToValue(hx: string): bigint;
+declare class Tx {
+    nonce: number;
+    gasPrice: number;
+    gasLimit: number;
+    to: Uint8Array;
+    value: bigint;
+    data: Uint8Array;
+    v: number;
+    r: Uint8Array;
+    s: Uint8Array;
+    chainId: number;
+    _signatureSet: boolean;
+    _workBuffer: ArrayBuffer;
+    _outBuffer: DataView;
+    _outBufferCursor: number;
+    constructor(chainId: number);
+    private serializeNumber;
+    private write;
+    serializeBytes(): Uint8Array;
+    canonicalOrder(): Uint8Array[];
+    serializeRLP(): Uint8Array;
+    message(): Uint8Array;
+    setSignature(r: Uint8Array, s: Uint8Array, v: number): void;
+    clearSignature(): void;
+}
+export { Tx, stringToValue, hexToValue, toValue, };

apps/cic-staff-client/src/assets/js/ethtx/tx.js

@@ -0,0 +1,121 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.toValue = exports.hexToValue = exports.stringToValue = exports.Tx = void 0;
+var hex_1 = require("./hex");
+var sha3_1 = require("sha3");
+var RLP = require('rlp');
+function isAddress(a) {
+    return a !== undefined && a.length == 20;
+}
+function toValue(n) {
+    return BigInt(n);
+}
+exports.toValue = toValue;
+function stringToValue(s) {
+    return BigInt(s);
+}
+exports.stringToValue = stringToValue;
+function hexToValue(hx) {
+    return BigInt(hex_1.add0x(hx));
+}
+exports.hexToValue = hexToValue;
+var Tx = /** @class */ (function () {
+    function Tx(chainId) {
+        this.chainId = chainId;
+        this.nonce = 0;
+        this.gasPrice = 0;
+        this.gasLimit = 0;
+        this.to = new Uint8Array(32);
+        this.data = new Uint8Array(0);
+        this.value = BigInt(0);
+        this._workBuffer = new ArrayBuffer(32);
+        this._outBuffer = new DataView(new ArrayBuffer(1024 * 1024));
+        this._outBufferCursor = 0;
+        this.clearSignature();
+    }
+    Tx.prototype.serializeNumber = function (n) {
+        var view = new DataView(this._workBuffer);
+        view.setBigUint64(0, BigInt(0));
+        view.setBigUint64(0, n);
+        var zeroOffset = 0;
+        for (zeroOffset = 0; zeroOffset < 8; zeroOffset++) {
+            if (view.getInt8(zeroOffset) > 0) {
+                break;
+            }
+        }
+        return new Uint8Array(this._workBuffer).slice(zeroOffset, 8);
+    };
+    Tx.prototype.write = function (data) {
+        var _this = this;
+        data.forEach(function (v) {
+            _this._outBuffer.setInt8(_this._outBufferCursor, v);
+            _this._outBufferCursor++;
+        });
+    };
+    Tx.prototype.serializeBytes = function () {
+        if (!isAddress(this.to)) {
+            throw new Error('invalid address');
+        }
+        var nonce = this.serializeNumber(BigInt(this.nonce));
+        this.write(nonce);
+        var gasPrice = this.serializeNumber(BigInt(this.gasPrice));
+        this.write(gasPrice);
+        var gasLimit = this.serializeNumber(BigInt(this.gasLimit));
+        this.write(gasLimit);
+        this.write(this.to);
+        var value = this.serializeNumber(this.value);
+        this.write(value);
+        this.write(this.data);
+        var v = this.serializeNumber(BigInt(this.v));
+        this.write(v);
+        this.write(this.r);
+        this.write(this.s);
+        return new Uint8Array(this._outBuffer.buffer).slice(0, this._outBufferCursor);
+    };
+    Tx.prototype.canonicalOrder = function () {
+        return [
+            this.serializeNumber(BigInt(this.nonce)),
+            this.serializeNumber(BigInt(this.gasPrice)),
+            this.serializeNumber(BigInt(this.gasLimit)),
+            this.to,
+            this.serializeNumber(this.value),
+            this.data,
+            this.serializeNumber(BigInt(this.v)),
+            this.r,
+            this.s,
+        ];
+    };
+    Tx.prototype.serializeRLP = function () {
+        return RLP.encode(this.canonicalOrder());
+    };
+    Tx.prototype.message = function () {
+        // TODO: Can we do without Buffer, pleeease?
+        var h = new sha3_1.Keccak(256);
+        var b = new Buffer(this.serializeRLP());
+        h.update(b);
+        return h.digest();
+    };
+    Tx.prototype.setSignature = function (r, s, v) {
+        if (this._signatureSet) {
+            throw new Error('Signature already set');
+        }
+        if (r.length != 32 || s.length != 32) {
+            throw new Error('Invalid signature length');
+        }
+        if (v < 0 || v > 3) {
+            throw new Error('Invalid recid');
+        }
+        this.r = r;
+        this.s = s;
+        this.v = (this.chainId * 2) + 35 + v;
+        this._signatureSet = true;
+    };
+    Tx.prototype.clearSignature = function () {
+        this.r = new Uint8Array(0);
+        this.s = new Uint8Array(0);
+        this.v = this.chainId;
+        this._signatureSet = false;
+    };
+    return Tx;
+}());
+exports.Tx = Tx;