move files out of scripts folder to their own dir
This commit is contained in:
@@ -87,7 +87,6 @@ COPY contract-migration/testdata/pgp testdata/pgp
|
||||
COPY contract-migration/sarafu_declaration.json sarafu_declaration.json
|
||||
COPY contract-migration/keystore keystore
|
||||
COPY contract-migration/envlist .
|
||||
COPY contract-migration/scripts scripts/
|
||||
|
||||
# A shared output dir for environment configs
|
||||
RUN mkdir -p /tmp/cic/config
|
||||
|
||||
@@ -1,260 +0,0 @@
|
||||
# DATA GENERATION TOOLS
|
||||
|
||||
This folder contains tools to generate and import test data.
|
||||
|
||||
## OVERVIEW
|
||||
|
||||
Three sets of tools are available, sorted by respective subdirectories.
|
||||
|
||||
* **eth**: Import using sovereign wallets.
|
||||
* **cic_eth**: Import using the `cic_eth` custodial engine.
|
||||
* **cic_ussd**: Import using the `cic_ussd` interface (backed by `cic_eth`)
|
||||
|
||||
Each of the modules include two main scripts:
|
||||
|
||||
* **import_users.py**: Registers all created accounts in the network
|
||||
* **import_balance.py**: Transfer an opening balance using an external keystore wallet
|
||||
|
||||
The balance script will sync with the blockchain, processing transactions and triggering actions when it finds. In its current version it does not keep track of any other state, so it will run indefinitly and needs You the Human to decide when it has done what it needs to do.
|
||||
|
||||
|
||||
In addition the following common tools are available:
|
||||
|
||||
* **create_import_users.py**: User creation script
|
||||
* **verify.py**: Import verification script
|
||||
* **cic_meta**: Metadata imports
|
||||
|
||||
|
||||
## REQUIREMENTS
|
||||
|
||||
A virtual environment for the python scripts is recommended. We know it works with `python 3.8.x`. Let us know if you run it successfully with other minor versions.
|
||||
|
||||
```
|
||||
python3 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
Install all requirements from the `requirements.txt` file:
|
||||
|
||||
`pip install --extra-index-url https://pip.grassrootseconomics.net:8433 -r requirements.txt`
|
||||
|
||||
|
||||
If you are importing metadata, also do ye olde:
|
||||
|
||||
`npm install`
|
||||
|
||||
|
||||
## HOW TO USE
|
||||
|
||||
### Step 1 - Data creation
|
||||
|
||||
Before running any of the imports, the user data to import has to be generated and saved to disk.
|
||||
|
||||
The script does not need any services to run.
|
||||
|
||||
Vanilla version:
|
||||
|
||||
`python create_import_users.py [--dir <datadir>] <number_of_users>`
|
||||
|
||||
If you want to use a `import_balance.py` script to add to the user's balance from an external address, use:
|
||||
|
||||
`python create_import_users.py --gift-threshold <max_units_to_send> [--dir <datadir>] <number_of_users>`
|
||||
|
||||
|
||||
### Step 2 - Services
|
||||
|
||||
Unless you know what you are doing, start with a clean slate, and execute (in the repository root):
|
||||
|
||||
`docker-compose down -v`
|
||||
|
||||
Then go through, in sequence:
|
||||
|
||||
#### Base requirements
|
||||
|
||||
If you are importing using `eth` and _not_ importing metadata, then the only service you need running in the cluster is:
|
||||
* eth
|
||||
|
||||
In all other cases you will _also_ need:
|
||||
* postgres
|
||||
* redis
|
||||
|
||||
|
||||
#### EVM provisions
|
||||
|
||||
This step is needed in *all* cases.
|
||||
|
||||
`RUN_MASK=1 docker-compose up contract-migration`
|
||||
|
||||
After this step is run, you can find top-level ethereum addresses (like the cic registry address, which you will need below) in `<repository_root>/service-configs/.env`
|
||||
|
||||
|
||||
#### Custodial provisions
|
||||
response_data = send_ussd_request(address, self.data_dir)
|
||||
state = response_data[:3]
|
||||
out = response_data[4:]
|
||||
m = '{} {}'.format(state, out[:7])
|
||||
if m != 'CON Welcome':
|
||||
raise VerifierError(response_data, 'ussd')
|
||||
This step is _only_ needed if you are importing using `cic_eth` or `cic_ussd`
|
||||
|
||||
`RUN_MASK=2 docker-compose up contract-migration`
|
||||
|
||||
|
||||
#### Custodial services
|
||||
|
||||
If importing using `cic_eth` or `cic_ussd` also run:
|
||||
* cic-eth-tasker
|
||||
* cic-eth-dispatcher
|
||||
* cic-eth-tracker
|
||||
* cic-eth-retrier
|
||||
|
||||
If importing using `cic_ussd` also run:
|
||||
* cic-user-tasker
|
||||
* cic-user-ussd-server
|
||||
* cic-notify-tasker
|
||||
|
||||
If metadata is to be imported, also run:
|
||||
* cic-meta-server
|
||||
|
||||
|
||||
|
||||
### Step 3 - User imports
|
||||
|
||||
If you did not change the docker-compose setup, your `eth_provider` the you need for the commands below will be `http://localhost:63545`.
|
||||
|
||||
Only run _one_ of the alternatives.
|
||||
|
||||
The keystore file used for transferring external opening balances tracker is relative to the directory you found this README in. Of course you can use a different wallet, but then you will have to provide it with tokens yourself (hint: `../reset.sh`)
|
||||
|
||||
All external balance transactions are saved in raw wire format in `<datadir>/txs`, with transaction hash as file name.
|
||||
|
||||
|
||||
|
||||
#### Alternative 1 - Sovereign wallet import - `eth`
|
||||
|
||||
|
||||
First, make a note of the **block height** before running anything.
|
||||
|
||||
To import, run to _completion_:
|
||||
|
||||
`python eth/import_users.py -v -c config -p <eth_provider> -r <cic_registry_address> -y ../keystore/UTC--2021-01-08T17-18-44.521011372Z--eb3907ecad74a0013c259d5874ae7f22dcbcc95c <datadir>`
|
||||
|
||||
After the script completes, keystore files for all generated accouts will be found in `<datadir>/keystore`, all with `foo` as password (would set it empty, but believe it or not some interfaces out there won't work unless you have one).
|
||||
|
||||
Then run:
|
||||
|
||||
`python eth/import_balance.py -v -c config -r <cic_registry_address> -p <eth_provider> --offset <block_height_at_start> -y ../keystore/UTC--2021-01-08T17-18-44.521011372Z--eb3907ecad74a0013c259d5874ae7f22dcbcc95c <datadir>`
|
||||
|
||||
|
||||
|
||||
#### Alternative 2 - Custodial engine import - `cic_eth`
|
||||
|
||||
Run in sequence, in first terminal:
|
||||
|
||||
`python cic_eth/import_balance.py -v -c config -p <eth_provider> -r <cic_registry_address> -y ../keystore/UTC--2021-01-08T17-18-44.521011372Z--eb3907ecad74a0013c259d5874ae7f22dcbcc95c --head out`
|
||||
|
||||
In another terminal:
|
||||
|
||||
`python cic_eth/import_users.py -v -c config --redis-host-callback <redis_hostname_in_docker> out`
|
||||
|
||||
The `redis_hostname_in_docker` value is the hostname required to reach the redis server from within the docker cluster, and should be `redis` if you left the docker-compose unchanged. The `import_users` script will receive the address of each newly created custodial account on a redis subscription fed by a callback task in the `cic_eth` account creation task chain.
|
||||
|
||||
|
||||
#### Alternative 3 - USSD import - `cic_ussd`
|
||||
|
||||
If you have previously run the `cic_ussd` import incompletely, it could be a good idea to purge the queue. If you have left docker-compose unchanged, `redis_url` should be `redis://localhost:63379`.
|
||||
|
||||
`celery -A cic_ussd.import_task purge -Q cic-import-ussd --broker <redis_url>`
|
||||
|
||||
Then, in sequence, run in first terminal:
|
||||
|
||||
`python cic_eth/import_balance.py -v -c config -p <eth_provider> -r <cic_registry_address> -y ../keystore/UTC--2021-01-08T17-18-44.521011372Z--eb3907ecad74a0013c259d5874ae7f22dcbcc95c out`
|
||||
|
||||
In second terminal:
|
||||
|
||||
`python cic_ussd/import_users.py -v -c config out`
|
||||
|
||||
|
||||
##### Importing pins and ussd data (optional)
|
||||
Once the user imports are complete the next step should be importing the user's pins and auxiliary ussd data. This can be done in 3 steps:
|
||||
|
||||
In one terminal run:
|
||||
|
||||
`python create_import_pins.py -c config -v --userdir <path to the users export dir tree> pinsdir <path to pin export dir tree>`
|
||||
|
||||
This script will recursively walk through all the directories defining user data in the users export directory and generate a csv file containing phone numbers and password hashes generated using fernet in a manner reflecting the nature of said hashes in the old system.
|
||||
This csv file will be stored in the pins export dir defined as the positional argument.
|
||||
|
||||
Once the creation of the pins file is complete, proceed to import the pins and ussd data as follows:
|
||||
|
||||
- To import the pins:
|
||||
|
||||
`python cic_ussd/import_pins.py -c config -v pinsdir <path to pin export dir tree>`
|
||||
|
||||
- To import ussd data:
|
||||
`python cic_ussd/import_ussd_data.py -c config -v userdir <path to the users export dir tree>`
|
||||
|
||||
The balance script is a celery task worker, and will not exit by itself in its current version. However, after it's done doing its job, you will find "reached nonce ... exiting" among the last lines of the log.
|
||||
|
||||
The connection parameters for the `cic-ussd-server` is currently _hardcoded_ in the `import_users.py` script file.
|
||||
|
||||
|
||||
### Step 4 - Metadata import (optional)
|
||||
|
||||
The metadata import scripts can be run at any time after step 1 has been completed.
|
||||
|
||||
|
||||
#### Importing user metadata
|
||||
|
||||
To import the main user metadata structs, run:
|
||||
|
||||
`node cic_meta/import_meta.js <datadir> <number_of_users>`
|
||||
|
||||
Monitors a folder for output from the `import_users.py` script, adding the metadata found to the `cic-meta` service.
|
||||
|
||||
If _number of users_ is omitted the script will run until manually interrupted.
|
||||
|
||||
|
||||
#### Importing phone pointer
|
||||
|
||||
`node cic_meta/import_meta_phone.js <datadir> <number_of_users>`
|
||||
|
||||
If you imported using `cic_ussd`, the phone pointer is _already added_ and this script will do nothing.
|
||||
|
||||
|
||||
### Step 5 - Verify
|
||||
|
||||
`python verify.py -v -c config -r <cic_registry_address> -p <eth_provider> <datadir>`
|
||||
|
||||
Included checks:
|
||||
* Private key is in cic-eth keystore
|
||||
* Address is in accounts index
|
||||
* Address has gas balance
|
||||
* Address has triggered the token faucet
|
||||
* Address has token balance matching the gift threshold
|
||||
* Personal metadata can be retrieved and has exact match
|
||||
* Phone pointer metadata can be retrieved and matches address
|
||||
* USSD menu response is initial state after registration
|
||||
|
||||
Checks can be selectively included and excluded. See `--help` for details.
|
||||
|
||||
Will output one line for each check, with name of check and number of errors found per check.
|
||||
|
||||
Should exit with code 0 if all input data is found in the respective services.
|
||||
|
||||
|
||||
## KNOWN ISSUES
|
||||
|
||||
- If the faucet disbursement is set to a non-zero amount, the balances will be off. The verify script needs to be improved to check the faucet amount.
|
||||
|
||||
- When the account callback in `cic_eth` fails, the `cic_eth/import_users.py` script will exit with a cryptic complaint concerning a `None` value.
|
||||
|
||||
- Sovereign import scripts use the same keystore, and running them simultaneously will mess up the transaction nonce sequence. Better would be to use two different keystore wallets so balance and users scripts can be run simultaneously.
|
||||
|
||||
- `pycrypto` and `pycryptodome` _have to be installed in that order_. If you get errors concerning `Crypto.KDF` then uninstall both and re-install in that order. Make sure you use the versions listed in `requirements.txt`. `pycryptodome` is a legacy dependency and will be removed as soon as possible.
|
||||
|
||||
- Sovereign import script is very slow because it's scrypt'ing keystore files for the accounts that it creates. An improvement would be optional and/or asynchronous keyfile generation.
|
||||
|
||||
- Running the balance script should be _optional_ in all cases, but is currently required in the case of `cic_ussd` because it is needed to generate the metadata. An improvement would be moving the task to `import_users.py`, for a different queue than the balance tx handler.
|
||||
|
||||
- `cic_ussd` imports is poorly implemented, and consumes a lot of resources. Therefore it takes a long time to complete. Reducing the amount of polls for the phone pointer would go a long way to improve it.
|
||||
@@ -1,314 +0,0 @@
|
||||
# standard imports
|
||||
import os
|
||||
import sys
|
||||
import logging
|
||||
import time
|
||||
import argparse
|
||||
import sys
|
||||
import re
|
||||
import hashlib
|
||||
import csv
|
||||
import json
|
||||
|
||||
# external imports
|
||||
import eth_abi
|
||||
import confini
|
||||
from hexathon import (
|
||||
strip_0x,
|
||||
add_0x,
|
||||
)
|
||||
from chainsyncer.backend.memory import MemBackend
|
||||
from chainsyncer.driver import HeadSyncer
|
||||
from chainlib.eth.connection import EthHTTPConnection
|
||||
from chainlib.eth.block import (
|
||||
block_latest,
|
||||
block_by_number,
|
||||
Block,
|
||||
)
|
||||
from chainlib.hash import keccak256_string_to_hex
|
||||
from chainlib.eth.address import to_checksum_address
|
||||
from chainlib.eth.gas import OverrideGasOracle
|
||||
from chainlib.eth.nonce import RPCNonceOracle
|
||||
from chainlib.eth.tx import TxFactory
|
||||
from chainlib.jsonrpc import jsonrpc_template
|
||||
from chainlib.eth.error import EthException
|
||||
from chainlib.chain import ChainSpec
|
||||
from chainlib.eth.constant import ZERO_ADDRESS
|
||||
from crypto_dev_signer.eth.signer import ReferenceSigner as EIP155Signer
|
||||
from crypto_dev_signer.keystore.dict import DictKeystore
|
||||
from cic_types.models.person import Person
|
||||
from eth_erc20 import ERC20
|
||||
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
config_dir = './config'
|
||||
|
||||
argparser = argparse.ArgumentParser(description='daemon that monitors transactions in new blocks')
|
||||
argparser.add_argument('-p', '--provider', dest='p', type=str, help='chain rpc provider address')
|
||||
argparser.add_argument('-y', '--key-file', dest='y', type=str, help='Ethereum keystore file to use for signing')
|
||||
argparser.add_argument('-c', type=str, default=config_dir, help='config root to use')
|
||||
argparser.add_argument('--old-chain-spec', type=str, dest='old_chain_spec', default='evm:oldchain:1', help='chain spec')
|
||||
argparser.add_argument('-i', '--chain-spec', type=str, dest='i', help='chain spec')
|
||||
argparser.add_argument('-r', '--registry-address', type=str, dest='r', help='CIC Registry address')
|
||||
argparser.add_argument('--token-symbol', default='GFT', type=str, dest='token_symbol', help='Token symbol to use for trnsactions')
|
||||
argparser.add_argument('--head', action='store_true', help='start at current block height (overrides --offset)')
|
||||
argparser.add_argument('--env-prefix', default=os.environ.get('CONFINI_ENV_PREFIX'), dest='env_prefix', type=str, help='environment prefix for variables to overwrite configuration')
|
||||
argparser.add_argument('-q', type=str, default='cic-eth', help='celery queue to submit transaction tasks to')
|
||||
argparser.add_argument('--offset', type=int, default=0, help='block offset to start syncer from')
|
||||
argparser.add_argument('-v', help='be verbose', action='store_true')
|
||||
argparser.add_argument('-vv', help='be more verbose', action='store_true')
|
||||
argparser.add_argument('user_dir', type=str, help='user export directory')
|
||||
args = argparser.parse_args(sys.argv[1:])
|
||||
|
||||
if args.v == True:
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
elif args.vv == True:
|
||||
logging.getLogger().setLevel(logging.DEBUG)
|
||||
|
||||
config_dir = os.path.join(args.c)
|
||||
os.makedirs(config_dir, 0o777, True)
|
||||
config = confini.Config(config_dir, args.env_prefix)
|
||||
config.process()
|
||||
# override args
|
||||
args_override = {
|
||||
'CIC_CHAIN_SPEC': getattr(args, 'i'),
|
||||
'ETH_PROVIDER': getattr(args, 'p'),
|
||||
'CIC_REGISTRY_ADDRESS': getattr(args, 'r'),
|
||||
}
|
||||
config.dict_override(args_override, 'cli flag')
|
||||
config.censor('PASSWORD', 'DATABASE')
|
||||
config.censor('PASSWORD', 'SSL')
|
||||
logg.debug('config loaded from {}:\n{}'.format(config_dir, config))
|
||||
|
||||
#app = celery.Celery(backend=config.get('CELERY_RESULT_URL'), broker=config.get('CELERY_BROKER_URL'))
|
||||
|
||||
signer_address = None
|
||||
keystore = DictKeystore()
|
||||
if args.y != None:
|
||||
logg.debug('loading keystore file {}'.format(args.y))
|
||||
signer_address = keystore.import_keystore_file(args.y)
|
||||
logg.debug('now have key for signer address {}'.format(signer_address))
|
||||
signer = EIP155Signer(keystore)
|
||||
|
||||
queue = args.q
|
||||
chain_str = config.get('CIC_CHAIN_SPEC')
|
||||
block_offset = 0
|
||||
if args.head:
|
||||
block_offset = -1
|
||||
else:
|
||||
block_offset = args.offset
|
||||
|
||||
chain_spec = ChainSpec.from_chain_str(chain_str)
|
||||
old_chain_spec_str = args.old_chain_spec
|
||||
old_chain_spec = ChainSpec.from_chain_str(old_chain_spec_str)
|
||||
|
||||
user_dir = args.user_dir # user_out_dir from import_users.py
|
||||
|
||||
token_symbol = args.token_symbol
|
||||
|
||||
|
||||
class Handler:
|
||||
|
||||
account_index_add_signature = keccak256_string_to_hex('add(address)')[:8]
|
||||
|
||||
def __init__(self, conn, chain_spec, user_dir, balances, token_address, signer, gas_oracle, nonce_oracle):
|
||||
self.token_address = token_address
|
||||
self.user_dir = user_dir
|
||||
self.balances = balances
|
||||
self.chain_spec = chain_spec
|
||||
self.tx_factory = ERC20(chain_spec, signer, gas_oracle, nonce_oracle)
|
||||
|
||||
|
||||
def name(self):
|
||||
return 'balance_handler'
|
||||
|
||||
|
||||
def filter(self, conn, block, tx, db_session):
|
||||
if tx.payload == None or len(tx.payload) == 0:
|
||||
logg.debug('no payload, skipping {}'.format(tx))
|
||||
return
|
||||
|
||||
if tx.payload[:8] == self.account_index_add_signature:
|
||||
recipient = eth_abi.decode_single('address', bytes.fromhex(tx.payload[-64:]))
|
||||
#original_address = to_checksum_address(self.addresses[to_checksum_address(recipient)])
|
||||
user_file = 'new/{}/{}/{}.json'.format(
|
||||
recipient[2:4].upper(),
|
||||
recipient[4:6].upper(),
|
||||
recipient[2:].upper(),
|
||||
)
|
||||
filepath = os.path.join(self.user_dir, user_file)
|
||||
o = None
|
||||
try:
|
||||
f = open(filepath, 'r')
|
||||
o = json.load(f)
|
||||
f.close()
|
||||
except FileNotFoundError:
|
||||
logg.error('no import record of address {}'.format(recipient))
|
||||
return
|
||||
u = Person.deserialize(o)
|
||||
original_address = u.identities[old_chain_spec.engine()]['{}:{}'.format(old_chain_spec.common_name(), old_chain_spec.network_id())][0]
|
||||
try:
|
||||
balance = self.balances[original_address]
|
||||
except KeyError as e:
|
||||
logg.error('balance get fail orig {} new {}'.format(original_address, recipient))
|
||||
return
|
||||
|
||||
# TODO: store token object in handler ,get decimals from there
|
||||
multiplier = 10**6
|
||||
balance_full = balance * multiplier
|
||||
logg.info('registered {} originally {} ({}) tx hash {} balance {}'.format(recipient, original_address, u, tx.hash, balance_full))
|
||||
|
||||
(tx_hash_hex, o) = self.tx_factory.transfer(self.token_address, signer_address, recipient, balance_full)
|
||||
logg.info('submitting erc20 transfer tx {} for recipient {}'.format(tx_hash_hex, recipient))
|
||||
r = conn.do(o)
|
||||
|
||||
tx_path = os.path.join(
|
||||
user_dir,
|
||||
'txs',
|
||||
strip_0x(tx_hash_hex),
|
||||
)
|
||||
f = open(tx_path, 'w')
|
||||
f.write(strip_0x(o['params'][0]))
|
||||
f.close()
|
||||
# except TypeError as e:
|
||||
# logg.warning('typerror {}'.format(e))
|
||||
# pass
|
||||
# except IndexError as e:
|
||||
# logg.warning('indexerror {}'.format(e))
|
||||
# pass
|
||||
# except EthException as e:
|
||||
# logg.error('send error {}'.format(e).ljust(200))
|
||||
#except KeyError as e:
|
||||
# logg.error('key record not found in imports: {}'.format(e).ljust(200))
|
||||
|
||||
|
||||
#class BlockGetter:
|
||||
#
|
||||
# def __init__(self, conn, gas_oracle, nonce_oracle, chain_spec):
|
||||
# self.conn = conn
|
||||
# self.tx_factory = ERC20(signer=signer, gas_oracle=gas_oracle, nonce_oracle=nonce_oracle, chain_id=chain_id)
|
||||
#
|
||||
#
|
||||
# def get(self, n):
|
||||
# o = block_by_number(n)
|
||||
# r = self.conn.do(o)
|
||||
# b = None
|
||||
# try:
|
||||
# b = Block(r)
|
||||
# except TypeError as e:
|
||||
# if r == None:
|
||||
# logg.debug('block not found {}'.format(n))
|
||||
# else:
|
||||
# logg.error('block retrieve error {}'.format(e))
|
||||
# return b
|
||||
|
||||
|
||||
def progress_callback(block_number, tx_index):
|
||||
sys.stdout.write(str(block_number).ljust(200) + "\n")
|
||||
|
||||
|
||||
|
||||
def main():
|
||||
global chain_str, block_offset, user_dir
|
||||
|
||||
conn = EthHTTPConnection(config.get('ETH_PROVIDER'))
|
||||
gas_oracle = OverrideGasOracle(conn=conn, limit=8000000)
|
||||
nonce_oracle = RPCNonceOracle(signer_address, conn)
|
||||
|
||||
# Get Token registry address
|
||||
txf = TxFactory(chain_spec, signer=signer, gas_oracle=gas_oracle, nonce_oracle=None)
|
||||
tx = txf.template(signer_address, config.get('CIC_REGISTRY_ADDRESS'))
|
||||
|
||||
registry_addressof_method = keccak256_string_to_hex('addressOf(bytes32)')[:8]
|
||||
data = add_0x(registry_addressof_method)
|
||||
data += eth_abi.encode_single('bytes32', b'TokenRegistry').hex()
|
||||
txf.set_code(tx, data)
|
||||
|
||||
o = jsonrpc_template()
|
||||
o['method'] = 'eth_call'
|
||||
o['params'].append(txf.normalize(tx))
|
||||
o['params'].append('latest')
|
||||
r = conn.do(o)
|
||||
token_index_address = to_checksum_address(eth_abi.decode_single('address', bytes.fromhex(strip_0x(r))))
|
||||
logg.info('found token index address {}'.format(token_index_address))
|
||||
|
||||
|
||||
# Get Sarafu token address
|
||||
tx = txf.template(signer_address, token_index_address)
|
||||
data = add_0x(registry_addressof_method)
|
||||
h = hashlib.new('sha256')
|
||||
h.update(token_symbol.encode('utf-8'))
|
||||
z = h.digest()
|
||||
data += eth_abi.encode_single('bytes32', z).hex()
|
||||
txf.set_code(tx, data)
|
||||
o = jsonrpc_template()
|
||||
o['method'] = 'eth_call'
|
||||
o['params'].append(txf.normalize(tx))
|
||||
o['params'].append('latest')
|
||||
r = conn.do(o)
|
||||
try:
|
||||
sarafu_token_address = to_checksum_address(eth_abi.decode_single('address', bytes.fromhex(strip_0x(r))))
|
||||
except ValueError as e:
|
||||
logg.critical('lookup failed for token {}: {}'.format(token_symbol, e))
|
||||
sys.exit(1)
|
||||
|
||||
if sarafu_token_address == ZERO_ADDRESS:
|
||||
raise KeyError('token address for symbol {} is zero'.format(token_symbol))
|
||||
|
||||
logg.info('found token address {}'.format(sarafu_token_address))
|
||||
|
||||
syncer_backend = MemBackend(chain_str, 0)
|
||||
|
||||
if block_offset == -1:
|
||||
o = block_latest()
|
||||
r = conn.do(o)
|
||||
block_offset = int(strip_0x(r), 16) + 1
|
||||
#
|
||||
# addresses = {}
|
||||
# f = open('{}/addresses.csv'.format(user_dir, 'r'))
|
||||
# while True:
|
||||
# l = f.readline()
|
||||
# if l == None:
|
||||
# break
|
||||
# r = l.split(',')
|
||||
# try:
|
||||
# k = r[0]
|
||||
# v = r[1].rstrip()
|
||||
# addresses[k] = v
|
||||
# sys.stdout.write('loading address mapping {} -> {}'.format(k, v).ljust(200) + "\r")
|
||||
# except IndexError as e:
|
||||
# break
|
||||
# f.close()
|
||||
|
||||
# TODO get decimals from token
|
||||
balances = {}
|
||||
f = open('{}/balances.csv'.format(user_dir, 'r'))
|
||||
remove_zeros = 10**6
|
||||
i = 0
|
||||
while True:
|
||||
l = f.readline()
|
||||
if l == None:
|
||||
break
|
||||
r = l.split(',')
|
||||
try:
|
||||
address = to_checksum_address(r[0])
|
||||
sys.stdout.write('loading balance {} {} {}'.format(i, address, r[1]).ljust(200) + "\r")
|
||||
except ValueError:
|
||||
break
|
||||
balance = int(int(r[1].rstrip()) / remove_zeros)
|
||||
balances[address] = balance
|
||||
i += 1
|
||||
|
||||
f.close()
|
||||
|
||||
syncer_backend.set(block_offset, 0)
|
||||
syncer = HeadSyncer(syncer_backend, block_callback=progress_callback)
|
||||
handler = Handler(conn, chain_spec, user_dir, balances, sarafu_token_address, signer, gas_oracle, nonce_oracle)
|
||||
syncer.add_filter(handler)
|
||||
syncer.loop(1, conn)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,254 +0,0 @@
|
||||
# standard imports
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import logging
|
||||
import argparse
|
||||
import uuid
|
||||
import datetime
|
||||
import time
|
||||
import phonenumbers
|
||||
from glob import glob
|
||||
|
||||
# third-party imports
|
||||
import redis
|
||||
import confini
|
||||
import celery
|
||||
from hexathon import (
|
||||
add_0x,
|
||||
strip_0x,
|
||||
)
|
||||
from chainlib.eth.address import to_checksum_address
|
||||
from cic_types.models.person import Person
|
||||
from cic_eth.api.api_task import Api
|
||||
from chainlib.chain import ChainSpec
|
||||
from cic_types.processor import generate_metadata_pointer
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
default_config_dir = '/usr/local/etc/cic'
|
||||
|
||||
argparser = argparse.ArgumentParser()
|
||||
argparser.add_argument('-c', type=str, default=default_config_dir, help='config file')
|
||||
argparser.add_argument('-i', '--chain-spec', dest='i', type=str, help='Chain specification string')
|
||||
argparser.add_argument('--old-chain-spec', type=str, dest='old_chain_spec', default='evm:oldchain:1', help='chain spec')
|
||||
argparser.add_argument('--redis-host', dest='redis_host', type=str, help='redis host to use for task submission')
|
||||
argparser.add_argument('--redis-port', dest='redis_port', type=int, help='redis host to use for task submission')
|
||||
argparser.add_argument('--redis-db', dest='redis_db', type=int, help='redis db to use for task submission and callback')
|
||||
argparser.add_argument('--redis-host-callback', dest='redis_host_callback', default='localhost', type=str, help='redis host to use for callback')
|
||||
argparser.add_argument('--redis-port-callback', dest='redis_port_callback', default=6379, type=int, help='redis port to use for callback')
|
||||
argparser.add_argument('--batch-size', dest='batch_size', default=100, type=int, help='burst size of sending transactions to node') # batch size should be slightly below cumulative gas limit worth, eg 80000 gas txs with 8000000 limit is a bit less than 100 batch size
|
||||
argparser.add_argument('--batch-delay', dest='batch_delay', default=2, type=int, help='seconds delay between batches')
|
||||
argparser.add_argument('--timeout', default=60.0, type=float, help='Callback timeout')
|
||||
argparser.add_argument('-q', type=str, default='cic-eth', help='Task queue')
|
||||
argparser.add_argument('-v', action='store_true', help='Be verbose')
|
||||
argparser.add_argument('-vv', action='store_true', help='Be more verbose')
|
||||
argparser.add_argument('user_dir', type=str, help='path to users export dir tree')
|
||||
args = argparser.parse_args()
|
||||
|
||||
if args.v:
|
||||
logg.setLevel(logging.INFO)
|
||||
elif args.vv:
|
||||
logg.setLevel(logging.DEBUG)
|
||||
|
||||
config_dir = args.c
|
||||
config = confini.Config(config_dir, os.environ.get('CONFINI_ENV_PREFIX'))
|
||||
config.process()
|
||||
args_override = {
|
||||
'CIC_CHAIN_SPEC': getattr(args, 'i'),
|
||||
'REDIS_HOST': getattr(args, 'redis_host'),
|
||||
'REDIS_PORT': getattr(args, 'redis_port'),
|
||||
'REDIS_DB': getattr(args, 'redis_db'),
|
||||
}
|
||||
config.dict_override(args_override, 'cli')
|
||||
celery_app = celery.Celery(broker=config.get('CELERY_BROKER_URL'), backend=config.get('CELERY_RESULT_URL'))
|
||||
|
||||
redis_host = config.get('REDIS_HOST')
|
||||
redis_port = config.get('REDIS_PORT')
|
||||
redis_db = config.get('REDIS_DB')
|
||||
r = redis.Redis(redis_host, redis_port, redis_db)
|
||||
|
||||
ps = r.pubsub()
|
||||
|
||||
user_new_dir = os.path.join(args.user_dir, 'new')
|
||||
os.makedirs(user_new_dir)
|
||||
|
||||
meta_dir = os.path.join(args.user_dir, 'meta')
|
||||
os.makedirs(meta_dir)
|
||||
|
||||
custom_dir = os.path.join(args.user_dir, 'custom')
|
||||
os.makedirs(custom_dir)
|
||||
os.makedirs(os.path.join(custom_dir, 'new'))
|
||||
os.makedirs(os.path.join(custom_dir, 'meta'))
|
||||
|
||||
phone_dir = os.path.join(args.user_dir, 'phone')
|
||||
os.makedirs(os.path.join(phone_dir, 'meta'))
|
||||
|
||||
user_old_dir = os.path.join(args.user_dir, 'old')
|
||||
os.stat(user_old_dir)
|
||||
|
||||
txs_dir = os.path.join(args.user_dir, 'txs')
|
||||
os.makedirs(txs_dir)
|
||||
|
||||
user_dir = args.user_dir
|
||||
|
||||
old_chain_spec = ChainSpec.from_chain_str(args.old_chain_spec)
|
||||
old_chain_str = str(old_chain_spec)
|
||||
|
||||
chain_spec = ChainSpec.from_chain_str(config.get('CIC_CHAIN_SPEC'))
|
||||
chain_str = str(chain_spec)
|
||||
|
||||
batch_size = args.batch_size
|
||||
batch_delay = args.batch_delay
|
||||
|
||||
|
||||
def register_eth(i, u):
|
||||
redis_channel = str(uuid.uuid4())
|
||||
ps.subscribe(redis_channel)
|
||||
#ps.get_message()
|
||||
api = Api(
|
||||
config.get('CIC_CHAIN_SPEC'),
|
||||
queue=args.q,
|
||||
callback_param='{}:{}:{}:{}'.format(args.redis_host_callback, args.redis_port_callback, redis_db, redis_channel),
|
||||
callback_task='cic_eth.callbacks.redis.redis',
|
||||
callback_queue=args.q,
|
||||
)
|
||||
t = api.create_account(register=True)
|
||||
logg.debug('register {} -> {}'.format(u, t))
|
||||
|
||||
while True:
|
||||
ps.get_message()
|
||||
m = ps.get_message(timeout=args.timeout)
|
||||
address = None
|
||||
if m == None:
|
||||
logg.debug('message timeout')
|
||||
return
|
||||
if m['type'] == 'subscribe':
|
||||
logg.debug('skipping subscribe message')
|
||||
continue
|
||||
try:
|
||||
r = json.loads(m['data'])
|
||||
address = r['result']
|
||||
break
|
||||
except Exception as e:
|
||||
if m == None:
|
||||
logg.critical('empty response from redis callback (did the service crash?) {}'.format(e))
|
||||
else:
|
||||
logg.critical('unexpected response from redis callback: {} {}'.format(m, e))
|
||||
sys.exit(1)
|
||||
logg.debug('[{}] register eth {} {}'.format(i, u, address))
|
||||
|
||||
return address
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
user_tags = {}
|
||||
f = open(os.path.join(user_dir, 'tags.csv'), 'r')
|
||||
while True:
|
||||
r = f.readline().rstrip()
|
||||
if len(r) == 0:
|
||||
break
|
||||
(old_address, tags_csv) = r.split(':')
|
||||
old_address = strip_0x(old_address)
|
||||
user_tags[old_address] = tags_csv.split(',')
|
||||
logg.debug('read tags {} for old address {}'.format(user_tags[old_address], old_address))
|
||||
|
||||
|
||||
i = 0
|
||||
j = 0
|
||||
for x in os.walk(user_old_dir):
|
||||
for y in x[2]:
|
||||
if y[len(y)-5:] != '.json':
|
||||
continue
|
||||
filepath = os.path.join(x[0], y)
|
||||
f = open(filepath, 'r')
|
||||
try:
|
||||
o = json.load(f)
|
||||
except json.decoder.JSONDecodeError as e:
|
||||
f.close()
|
||||
logg.error('load error for {}: {}'.format(y, e))
|
||||
continue
|
||||
f.close()
|
||||
logg.debug('deserializing {} {}'.format(filepath, o))
|
||||
u = Person.deserialize(o)
|
||||
|
||||
new_address = register_eth(i, u)
|
||||
if u.identities.get('evm') == None:
|
||||
u.identities['evm'] = {}
|
||||
sub_chain_str = '{}:{}'.format(chain_spec.common_name(), chain_spec.network_id())
|
||||
u.identities['evm'][sub_chain_str] = [new_address]
|
||||
|
||||
new_address_clean = strip_0x(new_address)
|
||||
filepath = os.path.join(
|
||||
user_new_dir,
|
||||
new_address_clean[:2].upper(),
|
||||
new_address_clean[2:4].upper(),
|
||||
new_address_clean.upper() + '.json',
|
||||
)
|
||||
os.makedirs(os.path.dirname(filepath), exist_ok=True)
|
||||
|
||||
o = u.serialize()
|
||||
f = open(filepath, 'w')
|
||||
f.write(json.dumps(o))
|
||||
f.close()
|
||||
|
||||
#fi.write('{},{}\n'.format(new_address, old_address))
|
||||
meta_key = generate_metadata_pointer(bytes.fromhex(new_address_clean), 'cic.person')
|
||||
meta_filepath = os.path.join(meta_dir, '{}.json'.format(new_address_clean.upper()))
|
||||
os.symlink(os.path.realpath(filepath), meta_filepath)
|
||||
|
||||
phone_object = phonenumbers.parse(u.tel)
|
||||
phone = phonenumbers.format_number(phone_object, phonenumbers.PhoneNumberFormat.E164)
|
||||
meta_phone_key = generate_metadata_pointer(phone.encode('utf-8'), ':cic.phone')
|
||||
meta_phone_filepath = os.path.join(phone_dir, 'meta', meta_phone_key)
|
||||
|
||||
filepath = os.path.join(
|
||||
phone_dir,
|
||||
'new',
|
||||
meta_phone_key[:2].upper(),
|
||||
meta_phone_key[2:4].upper(),
|
||||
meta_phone_key.upper(),
|
||||
)
|
||||
os.makedirs(os.path.dirname(filepath), exist_ok=True)
|
||||
|
||||
f = open(filepath, 'w')
|
||||
f.write(to_checksum_address(new_address_clean))
|
||||
f.close()
|
||||
|
||||
os.symlink(os.path.realpath(filepath), meta_phone_filepath)
|
||||
|
||||
|
||||
# custom data
|
||||
custom_key = generate_metadata_pointer(phone.encode('utf-8'), ':cic.custom')
|
||||
custom_filepath = os.path.join(custom_dir, 'meta', custom_key)
|
||||
|
||||
filepath = os.path.join(
|
||||
custom_dir,
|
||||
'new',
|
||||
custom_key[:2].upper(),
|
||||
custom_key[2:4].upper(),
|
||||
custom_key.upper() + '.json',
|
||||
)
|
||||
os.makedirs(os.path.dirname(filepath), exist_ok=True)
|
||||
|
||||
sub_old_chain_str = '{}:{}'.format(old_chain_spec.common_name(), old_chain_spec.network_id())
|
||||
f = open(filepath, 'w')
|
||||
k = u.identities['evm'][sub_old_chain_str][0]
|
||||
tag_data = {'tags': user_tags[strip_0x(k)]}
|
||||
f.write(json.dumps(tag_data))
|
||||
f.close()
|
||||
|
||||
os.symlink(os.path.realpath(filepath), custom_filepath)
|
||||
|
||||
|
||||
i += 1
|
||||
sys.stdout.write('imported {} {}'.format(i, u).ljust(200) + "\r")
|
||||
|
||||
j += 1
|
||||
if j == batch_size:
|
||||
time.sleep(batch_delay)
|
||||
j = 0
|
||||
|
||||
#fi.close()
|
||||
@@ -1,50 +0,0 @@
|
||||
# standard imports
|
||||
import sys
|
||||
import os
|
||||
import logging
|
||||
import json
|
||||
|
||||
# external imports
|
||||
import celery
|
||||
import confini
|
||||
|
||||
# local imports
|
||||
from cic_eth.api import Api
|
||||
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
logg = logging.getLogger()
|
||||
|
||||
|
||||
script_dir = os.path.realpath(os.path.dirname(__file__))
|
||||
config_dir = os.path.join(script_dir, '..', 'config')
|
||||
config = confini.Config(config_dir, os.environ.get('CONFINI_ENV_PREFIX'))
|
||||
config.process()
|
||||
|
||||
celery_app = celery.Celery(broker=config.get('CELERY_BROKER_URL'), backend=config.get('CELERY_RESULT_URL'), result_extended=True)
|
||||
|
||||
|
||||
class Fmtr(celery.utils.graph.GraphFormatter):
|
||||
|
||||
def label(self, obj):
|
||||
super(Fmtr, self).label(obj)
|
||||
if obj != None:
|
||||
if obj.name == None:
|
||||
raise RuntimeError('task name is not defined. Did you run celery with result_extended=True?')
|
||||
return obj.name
|
||||
|
||||
|
||||
def main():
|
||||
api = Api(
|
||||
config.get('CIC_CHAIN_SPEC'),
|
||||
queue='cic-eth',
|
||||
#callback_param='{}:{}:{}:{}'.format(args.redis_host_callback, args.redis_port_callback, redis_db, redis_channel),
|
||||
#callback_task='cic_eth.callbacks.redis.redis',
|
||||
#callback_queue=args.q,
|
||||
)
|
||||
t = api.create_account(register=False)
|
||||
t.get_leaf()
|
||||
t.build_graph(intermediate=True, formatter=Fmtr()).to_dot(sys.stdout)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,417 +0,0 @@
|
||||
# standard imports
|
||||
import logging
|
||||
import json
|
||||
import uuid
|
||||
import importlib
|
||||
import random
|
||||
import copy
|
||||
from argparse import RawTextHelpFormatter
|
||||
|
||||
# external imports
|
||||
import redis
|
||||
from cic_eth.api.api_task import Api
|
||||
|
||||
logg = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def add_args(argparser):
|
||||
"""Parse script specific command line arguments
|
||||
|
||||
:param argparser: Top-level argument parser
|
||||
:type argparser: argparse.ArgumentParser
|
||||
"""
|
||||
argparser.formatter_class = formatter_class=RawTextHelpFormatter
|
||||
argparser.add_argument('--redis-host-callback', dest='redis_host_callback', default='localhost', type=str, help='redis host to use for callback')
|
||||
argparser.add_argument('--redis-port-callback', dest='redis_port_callback', default=6379, type=int, help='redis port to use for callback')
|
||||
argparser.add_argument('--batch-size', dest='batch_size', default=10, type=int, help='number of events to process simultaneously')
|
||||
argparser.description = """Generates traffic on the cic network using dynamically loaded modules as event sources
|
||||
|
||||
"""
|
||||
return argparser
|
||||
|
||||
|
||||
class TrafficItem:
|
||||
"""Represents a single item of traffic meta that will be processed by a traffic generation method
|
||||
|
||||
The traffic generation module passed in the argument must implement a method "do" with interface conforming to local.noop_traffic.do.
|
||||
|
||||
:param item: Traffic generation module.
|
||||
:type item: function
|
||||
"""
|
||||
def __init__(self, item):
|
||||
self.method = item.do
|
||||
self.uuid = uuid.uuid4()
|
||||
self.ext = None
|
||||
self.result = None
|
||||
self.sender = None
|
||||
self.recipient = None
|
||||
self.source_token = None
|
||||
self.destination_token = None
|
||||
self.source_value = 0
|
||||
|
||||
|
||||
def __str__(self):
|
||||
return 'traffic item method {} uuid {}'.format(self.method, self.uuid)
|
||||
|
||||
|
||||
class TrafficRouter:
|
||||
"""Holds and selects from the collection of traffic generator modules that will be used for the execution.
|
||||
|
||||
:params batch_size: Amount of simultaneous traffic items that can simultanously be in flight.
|
||||
:type batch_size: number
|
||||
:raises ValueError: If batch size is zero of negative
|
||||
"""
|
||||
def __init__(self, batch_size=1):
|
||||
if batch_size < 1:
|
||||
raise ValueError('batch size cannot be 0')
|
||||
self.items = []
|
||||
self.weights = []
|
||||
self.total_weights = 0
|
||||
self.batch_size = batch_size
|
||||
self.reserved = {}
|
||||
self.reserved_count = 0
|
||||
self.traffic = {}
|
||||
|
||||
|
||||
def add(self, item, weight):
|
||||
"""Add a traffic generator module to the list of modules to choose between for traffic item exectuion.
|
||||
|
||||
The probability that a module will be chosen for any single item is the ratio between the weight parameter and the accumulated weights for all items.
|
||||
|
||||
See local.noop for which criteria the generator module must fulfill.
|
||||
|
||||
:param item: Qualified class path to traffic generator module. Will be dynamically loaded.
|
||||
:type item: str
|
||||
:param weight: Selection probability weight
|
||||
:type weight: number
|
||||
:raises ModuleNotFound: Invalid item argument
|
||||
"""
|
||||
self.weights.append(self.total_weights)
|
||||
self.total_weights += weight
|
||||
m = importlib.import_module(item)
|
||||
self.items.append(m)
|
||||
|
||||
|
||||
def reserve(self):
|
||||
"""Selects the module to be used to execute the next traffic item, using the provided weights.
|
||||
|
||||
If the current number of calls to "reserve" without corresponding calls to "release" equals the set batch size limit, None will be returned. The calling code should allow a short grace period before trying the call again.
|
||||
:raises ValueError: No items have been added
|
||||
:returns: A traffic item with the selected module method as the method property.
|
||||
:rtype: TrafficItem|None
|
||||
"""
|
||||
if len(self.items) == 0:
|
||||
raise ValueError('Add at least one item first')
|
||||
|
||||
if len(self.reserved) == self.batch_size:
|
||||
return None
|
||||
|
||||
n = random.randint(0, self.total_weights)
|
||||
item = self.items[0]
|
||||
for i in range(len(self.weights)):
|
||||
if n <= self.weights[i]:
|
||||
item = self.items[i]
|
||||
break
|
||||
|
||||
ti = TrafficItem(item)
|
||||
self.reserved[ti.uuid] = ti
|
||||
return ti
|
||||
|
||||
|
||||
def release(self, traffic_item):
|
||||
"""Releases the traffic item from the list of simultaneous traffic items in flight.
|
||||
|
||||
:param traffic_item: Traffic item
|
||||
:type traffic_item: TrafficItem
|
||||
"""
|
||||
del self.reserved[traffic_item.uuid]
|
||||
|
||||
|
||||
def apply_import_dict(self, keys, dct):
|
||||
"""Convenience method to add traffic generator modules from a dictionary.
|
||||
|
||||
:param keys: Keys in dictionary to add
|
||||
:type keys: list of str
|
||||
:param dct: Dictionary to choose module strings from
|
||||
:type dct: dict
|
||||
:raises ModuleNotFoundError: If one of the module strings refer to an invalid module.
|
||||
"""
|
||||
# parse traffic items
|
||||
for k in keys:
|
||||
if len(k) > 8 and k[:8] == 'TRAFFIC_':
|
||||
v = int(dct.get(k))
|
||||
self.add(k[8:].lower(), v)
|
||||
logg.debug('found traffic item {} weight {}'.format(k, v))
|
||||
|
||||
|
||||
# TODO: This will not work well with big networks. The provisioner should use lazy loading and LRU instead.
|
||||
class TrafficProvisioner:
|
||||
"""Loads metadata necessary for traffic item execution.
|
||||
|
||||
Instantiation will by default trigger retrieval of accounts and tokens on the network.
|
||||
|
||||
It will also populate the aux property of the instance with the values from the static aux parameter template.
|
||||
"""
|
||||
|
||||
oracles = {
|
||||
'account': None,
|
||||
'token': None,
|
||||
}
|
||||
"""Data oracles to be used for traffic item generation"""
|
||||
default_aux = {
|
||||
}
|
||||
"""Aux parameter template to be passed to the traffic generator module"""
|
||||
|
||||
|
||||
def __init__(self):
|
||||
self.tokens = self.oracles['token'].get_tokens()
|
||||
self.accounts = self.oracles['account'].get_accounts()
|
||||
self.aux = copy.copy(self.default_aux)
|
||||
self.__balances = {}
|
||||
for a in self.accounts:
|
||||
self.__balances[a] = {}
|
||||
|
||||
|
||||
# Caches a single address' balance of a single token
|
||||
def __cache_balance(self, holder_address, token, value):
|
||||
if self.__balances.get(holder_address) == None:
|
||||
self.__balances[holder_address] = {}
|
||||
self.__balances[holder_address][token] = value
|
||||
logg.debug('setting cached balance of {} token {} to {}'.format(holder_address, token, value))
|
||||
|
||||
|
||||
def add_aux(self, k, v):
|
||||
"""Add a key-value pair to the aux parameter list.
|
||||
|
||||
Does not protect existing entries from being overwritten.
|
||||
|
||||
:param k: Key
|
||||
:type k: str
|
||||
:param v: Value
|
||||
:type v: any
|
||||
"""
|
||||
logg.debug('added {} = {} to traffictasker'.format(k, v))
|
||||
self.aux[k] = v
|
||||
|
||||
|
||||
# TODO: Balance list type should perhaps be a class (provided by cic-eth package) due to its complexity.
|
||||
def balances(self, refresh_accounts=None):
|
||||
"""Retrieves all token balances for the given account list.
|
||||
|
||||
If refresh_accounts is not None, the balance values for the given accounts will be retrieved from upstream. If the argument is an empty list, the balances will be updated for all tokens of all ccounts. If there are many accounts and/or tokens, this may be a VERY EXPENSIVE OPERATION. The "balance" method can be used instead to update individual account/token pair balances.
|
||||
|
||||
:param accounts: List of accounts to refresh balances for.
|
||||
:type accounts: list of str, 0x-hex
|
||||
:returns: Dict of dict of dicts; v[accounts][token] = {balance_types}
|
||||
:rtype: dict
|
||||
"""
|
||||
if refresh_accounts != None:
|
||||
accounts = refresh_accounts
|
||||
if len(accounts) == 0:
|
||||
accounts = self.accounts
|
||||
for account in accounts:
|
||||
for token in self.tokens:
|
||||
value = self.balance(account, token)
|
||||
self.__cache_balance(account, token.symbol(), value)
|
||||
logg.debug('balance sender {} token {} = {}'.format(account, token, value))
|
||||
else:
|
||||
logg.debug('returning cached balances')
|
||||
|
||||
return self.__balances
|
||||
|
||||
|
||||
# TODO: use proper redis callback
|
||||
def balance(self, account, token):
|
||||
"""Update balance for a single token of a single account from upstream.
|
||||
|
||||
The balance will be the spendable balance at the time of the call. This value may be less than the balance reported by the consensus network, if a previous outgoing transaction is still pending in the network or the custodial system queue.
|
||||
|
||||
:param account: Account to update
|
||||
:type account: str, 0x-hex
|
||||
:param token: Token to update balance for
|
||||
:type token: cic_registry.token.Token
|
||||
:returns: Updated balance
|
||||
:rtype: complex balance dict
|
||||
"""
|
||||
api = Api(
|
||||
str(self.aux['chain_spec']),
|
||||
queue=self.aux['api_queue'],
|
||||
#callback_param='{}:{}:{}:{}'.format(aux['redis_host_callback'], aux['redis_port_callback'], aux['redis_db'], aux['redis_channel']),
|
||||
#callback_task='cic_eth.callbacks.redis.redis',
|
||||
#callback_queue=queue,
|
||||
)
|
||||
t = api.balance(account, token.symbol())
|
||||
r = t.get()
|
||||
for c in t.collect():
|
||||
r = c[1]
|
||||
assert t.successful()
|
||||
#return r[0]['balance_network'] - r[0]['balance_outgoing']
|
||||
return r[0]
|
||||
|
||||
|
||||
def update_balance(self, account, token, value):
|
||||
"""Manually set a token balance for an account.
|
||||
|
||||
:param account: Account to update
|
||||
:type account: str, 0x-hex
|
||||
:param token: Token to update balance for
|
||||
:type token: cic_registry.token.Token
|
||||
:param value: Balance value to set
|
||||
:type value: number
|
||||
:returns: Balance value (unchanged)
|
||||
:rtype: complex balance dict
|
||||
"""
|
||||
self.__cache_balance(account, token.symbol(), value)
|
||||
return value
|
||||
|
||||
|
||||
# TODO: Abstract redis with a generic pubsub adapter
|
||||
class TrafficSyncHandler:
|
||||
"""Encapsulates callback methods required by the chain syncer.
|
||||
|
||||
This implementation uses a redis subscription as backend to retrieve results from asynchronously executed tasks.
|
||||
|
||||
:param config: Configuration of current top-level execution
|
||||
:type config: object with dict get interface
|
||||
:param traffic_router: Traffic router instance to use for the syncer session.
|
||||
:type traffic_router: TrafficRouter
|
||||
:raises Exception: Any Exception redis may raise on connection attempt.
|
||||
"""
|
||||
def __init__(self, config, traffic_router):
|
||||
self.traffic_router = traffic_router
|
||||
self.redis_channel = str(uuid.uuid4())
|
||||
self.pubsub = self.__connect_redis(self.redis_channel, config)
|
||||
self.traffic_items = {}
|
||||
self.config = config
|
||||
self.init = False
|
||||
|
||||
|
||||
# connects to redis
|
||||
def __connect_redis(self, redis_channel, config):
|
||||
r = redis.Redis(config.get('REDIS_HOST'), config.get('REDIS_PORT'), config.get('REDIS_DB'))
|
||||
redis_pubsub = r.pubsub()
|
||||
redis_pubsub.subscribe(redis_channel)
|
||||
logg.debug('redis connected on channel {}'.format(redis_channel))
|
||||
return redis_pubsub
|
||||
|
||||
|
||||
# TODO: This method is too long, split up
|
||||
# TODO: This method will not yet cache balances for newly created accounts
|
||||
def refresh(self, block_number, tx_index):
|
||||
"""Traffic method and item execution driver to be called on every loop execution of the chain syncer.
|
||||
|
||||
Implements the signature required by callbacks called from chainsyncer.driver.loop.
|
||||
|
||||
:param block_number: Syncer block height at time of call.
|
||||
:type block_number: number
|
||||
:param tx_index: Syncer block transaction index at time of call.
|
||||
:type tx_index: number
|
||||
"""
|
||||
traffic_provisioner = TrafficProvisioner()
|
||||
traffic_provisioner.add_aux('redis_channel', self.redis_channel)
|
||||
|
||||
refresh_accounts = None
|
||||
# Note! This call may be very expensive if there are a lot of accounts and/or tokens on the network
|
||||
if not self.init:
|
||||
refresh_accounts = traffic_provisioner.accounts
|
||||
balances = traffic_provisioner.balances(refresh_accounts=refresh_accounts)
|
||||
self.init = True
|
||||
|
||||
if len(traffic_provisioner.tokens) == 0:
|
||||
logg.error('patiently waiting for at least one registered token...')
|
||||
return
|
||||
|
||||
logg.debug('executing handler refresh with accounts {}'.format(traffic_provisioner.accounts))
|
||||
logg.debug('executing handler refresh with tokens {}'.format(traffic_provisioner.tokens))
|
||||
|
||||
sender_indices = [*range(0, len(traffic_provisioner.accounts))]
|
||||
# TODO: only get balances for the selection that we will be generating for
|
||||
|
||||
while True:
|
||||
traffic_item = self.traffic_router.reserve()
|
||||
if traffic_item == None:
|
||||
logg.debug('no traffic_items left to reserve {}'.format(traffic_item))
|
||||
break
|
||||
|
||||
# TODO: temporary selection
|
||||
token_pair = (
|
||||
traffic_provisioner.tokens[0],
|
||||
traffic_provisioner.tokens[0],
|
||||
)
|
||||
sender_index_index = random.randint(0, len(sender_indices)-1)
|
||||
sender_index = sender_indices[sender_index_index]
|
||||
sender = traffic_provisioner.accounts[sender_index]
|
||||
#balance_full = balances[sender][token_pair[0].symbol()]
|
||||
if len(sender_indices) == 1:
|
||||
sender_indices[m] = sender_sender_indices[len(senders)-1]
|
||||
sender_indices = sender_indices[:len(sender_indices)-1]
|
||||
|
||||
balance_full = traffic_provisioner.balance(sender, token_pair[0])
|
||||
|
||||
recipient_index = random.randint(0, len(traffic_provisioner.accounts)-1)
|
||||
recipient = traffic_provisioner.accounts[recipient_index]
|
||||
|
||||
logg.debug('trigger item {} tokens {} sender {} recipient {} balance {}')
|
||||
(e, t, balance_result,) = traffic_item.method(
|
||||
token_pair,
|
||||
sender,
|
||||
recipient,
|
||||
balance_full,
|
||||
traffic_provisioner.aux,
|
||||
block_number,
|
||||
tx_index,
|
||||
)
|
||||
traffic_provisioner.update_balance(sender, token_pair[0], balance_result)
|
||||
sender_indices.append(recipient_index)
|
||||
|
||||
if e != None:
|
||||
logg.info('failed {}: {}'.format(str(traffic_item), e))
|
||||
self.traffic_router.release(traffic_item)
|
||||
continue
|
||||
|
||||
if t == None:
|
||||
logg.info('traffic method {} completed immediately')
|
||||
self.traffic_router.release(traffic_item)
|
||||
traffic_item.ext = t
|
||||
self.traffic_items[traffic_item.ext] = traffic_item
|
||||
|
||||
|
||||
while True:
|
||||
m = self.pubsub.get_message(timeout=0.1)
|
||||
if m == None:
|
||||
break
|
||||
logg.debug('redis message {}'.format(m))
|
||||
if m['type'] == 'message':
|
||||
message_data = json.loads(m['data'])
|
||||
uu = message_data['root_id']
|
||||
match_item = self.traffic_items[uu]
|
||||
self.traffic_router.release(match_item)
|
||||
logg.debug('>>>>>>>>>>>>>>>>>>> match item {} {} {}'.format(match_item, match_item.result, dir(match_item)))
|
||||
if message_data['status'] != 0:
|
||||
logg.error('task item {} failed with error code {}'.format(match_item, message_data['status']))
|
||||
else:
|
||||
match_item.result = message_data['result']
|
||||
logg.debug('got callback result: {}'.format(match_item))
|
||||
|
||||
|
||||
def name(self):
|
||||
"""Returns the common name for the syncer callback implementation. Required by the chain syncer.
|
||||
"""
|
||||
return 'traffic_item_handler'
|
||||
|
||||
|
||||
def filter(self, conn, block, tx, db_session):
|
||||
"""Callback for every transaction found in a block. Required by the chain syncer.
|
||||
|
||||
Currently performs no operation.
|
||||
|
||||
:param conn: A HTTPConnection object to the chain rpc provider.
|
||||
:type conn: chainlib.eth.rpc.HTTPConnection
|
||||
:param block: The block object of current transaction
|
||||
:type block: chainlib.eth.block.Block
|
||||
:param tx: The block transaction object
|
||||
:type tx: chainlib.eth.tx.Tx
|
||||
:param db_session: Syncer backend database session
|
||||
:type db_session: SQLAlchemy.Session
|
||||
"""
|
||||
logg.debug('handler get {}'.format(tx))
|
||||
@@ -1,8 +0,0 @@
|
||||
from . import (
|
||||
log,
|
||||
argparse,
|
||||
config,
|
||||
signer,
|
||||
rpc,
|
||||
registry,
|
||||
)
|
||||
@@ -1,73 +0,0 @@
|
||||
# standard imports
|
||||
import logging
|
||||
import argparse
|
||||
import os
|
||||
import sys
|
||||
|
||||
default_config_dir = os.environ.get('CONFINI_DIR')
|
||||
full_template = {
|
||||
# (long arg and key name, short var, type, default, help,)
|
||||
'provider': ('p', str, None, 'RPC provider url',),
|
||||
'registry_address': ('r', str, None, 'CIC registry address',),
|
||||
'keystore_file': ('y', str, None, 'Keystore file',),
|
||||
'config_dir': ('c', str, default_config_dir, 'Configuration directory',),
|
||||
'queue': ('q', str, 'cic-eth', 'Celery task queue',),
|
||||
'chain_spec': ('i', str, None, 'Chain spec string',),
|
||||
'abi_dir': (None, str, None, 'Smart contract ABI search path',),
|
||||
'env_prefix': (None, str, os.environ.get('CONFINI_ENV_PREFIX'), 'Environment prefix for variables to overwrite configuration',),
|
||||
}
|
||||
default_include_args = [
|
||||
'config_dir',
|
||||
'provider',
|
||||
'env_prefix',
|
||||
]
|
||||
|
||||
sub = None
|
||||
|
||||
def create(caller_dir, include_args=default_include_args):
|
||||
|
||||
argparser = argparse.ArgumentParser()
|
||||
|
||||
for k in include_args:
|
||||
a = full_template[k]
|
||||
long_flag = '--' + k.replace('_', '-')
|
||||
short_flag = None
|
||||
dest = None
|
||||
if a[0] != None:
|
||||
short_flag = '-' + a[0]
|
||||
dest = a[0]
|
||||
else:
|
||||
dest = k
|
||||
default = a[2]
|
||||
if default == None and k == 'config_dir':
|
||||
default = os.path.join(caller_dir, 'config')
|
||||
|
||||
if short_flag == None:
|
||||
argparser.add_argument(long_flag, dest=dest, type=a[1], default=default, help=a[3])
|
||||
else:
|
||||
argparser.add_argument(short_flag, long_flag, dest=dest, type=a[1], default=default, help=a[3])
|
||||
|
||||
argparser.add_argument('-v', action='store_true', help='Be verbose')
|
||||
argparser.add_argument('-vv', action='store_true', help='Be more verbose')
|
||||
|
||||
return argparser
|
||||
|
||||
|
||||
def add(argparser, processor, name, description=None):
|
||||
processor(argparser)
|
||||
|
||||
return argparser
|
||||
|
||||
|
||||
def parse(argparser, logger=None):
|
||||
|
||||
args = argparser.parse_args(sys.argv[1:])
|
||||
|
||||
# handle logging input
|
||||
if logger != None:
|
||||
if args.vv:
|
||||
logger.setLevel(logging.DEBUG)
|
||||
elif args.v:
|
||||
logger.setLevel(logging.INFO)
|
||||
|
||||
return args
|
||||
@@ -1,39 +0,0 @@
|
||||
# external imports
|
||||
import logging
|
||||
import confini
|
||||
|
||||
logg = logging.getLogger(__name__)
|
||||
|
||||
default_arg_overrides = {
|
||||
'abi_dir': 'ETH_ABI_DIR',
|
||||
'p': 'ETH_PROVIDER',
|
||||
'i': 'CIC_CHAIN_SPEC',
|
||||
'r': 'CIC_REGISTRY_ADDRESS',
|
||||
}
|
||||
|
||||
|
||||
def override(config, override_dict, label):
|
||||
config.dict_override(override_dict, label)
|
||||
config.validate()
|
||||
return config
|
||||
|
||||
|
||||
def create(config_dir, args, env_prefix=None, arg_overrides=default_arg_overrides):
|
||||
# handle config input
|
||||
config = confini.Config(config_dir, env_prefix)
|
||||
config.process()
|
||||
if arg_overrides != None and args != None:
|
||||
override_dict = {}
|
||||
for k in arg_overrides:
|
||||
v = getattr(args, k)
|
||||
if v != None:
|
||||
override_dict[arg_overrides[k]] = v
|
||||
config = override(config, override_dict, 'args')
|
||||
else:
|
||||
config.validate()
|
||||
|
||||
return config
|
||||
|
||||
|
||||
def log(config):
|
||||
logg.debug('config loaded:\n{}'.format(config))
|
||||
@@ -1,18 +0,0 @@
|
||||
# standard imports
|
||||
import logging
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
|
||||
default_mutelist = [
|
||||
'urllib3',
|
||||
'websockets.protocol',
|
||||
'web3.RequestManager',
|
||||
'web3.providers.WebsocketProvider',
|
||||
'web3.providers.HTTPProvider',
|
||||
]
|
||||
|
||||
def create(name=None, mutelist=default_mutelist):
|
||||
logg = logging.getLogger(name)
|
||||
for m in mutelist:
|
||||
logging.getLogger(m).setLevel(logging.CRITICAL)
|
||||
return logg
|
||||
@@ -1,86 +0,0 @@
|
||||
# standard imports
|
||||
import logging
|
||||
import copy
|
||||
|
||||
# external imports
|
||||
from cic_registry import CICRegistry
|
||||
from eth_token_index import TokenUniqueSymbolIndex
|
||||
from eth_accounts_index import AccountRegistry
|
||||
from chainlib.chain import ChainSpec
|
||||
from cic_registry.chain import ChainRegistry
|
||||
from cic_registry.helper.declarator import DeclaratorOracleAdapter
|
||||
|
||||
logg = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class TokenOracle:
|
||||
|
||||
def __init__(self, conn, chain_spec, registry):
|
||||
self.tokens = []
|
||||
self.chain_spec = chain_spec
|
||||
self.registry = registry
|
||||
|
||||
token_registry_contract = CICRegistry.get_contract(chain_spec, 'TokenRegistry', 'Registry')
|
||||
self.getter = TokenUniqueSymbolIndex(conn, token_registry_contract.address())
|
||||
|
||||
|
||||
def get_tokens(self):
|
||||
token_count = self.getter.count()
|
||||
if token_count == len(self.tokens):
|
||||
return self.tokens
|
||||
|
||||
for i in range(len(self.tokens), token_count):
|
||||
token_address = self.getter.get_index(i)
|
||||
t = self.registry.get_address(self.chain_spec, token_address)
|
||||
token_symbol = t.symbol()
|
||||
self.tokens.append(t)
|
||||
|
||||
logg.debug('adding token idx {} symbol {} address {}'.format(i, token_symbol, token_address))
|
||||
|
||||
return copy.copy(self.tokens)
|
||||
|
||||
|
||||
class AccountsOracle:
|
||||
|
||||
def __init__(self, conn, chain_spec, registry):
|
||||
self.accounts = []
|
||||
self.chain_spec = chain_spec
|
||||
self.registry = registry
|
||||
|
||||
accounts_registry_contract = CICRegistry.get_contract(chain_spec, 'AccountRegistry', 'Registry')
|
||||
self.getter = AccountRegistry(conn, accounts_registry_contract.address())
|
||||
|
||||
|
||||
def get_accounts(self):
|
||||
accounts_count = self.getter.count()
|
||||
if accounts_count == len(self.accounts):
|
||||
return self.accounts
|
||||
|
||||
for i in range(len(self.accounts), accounts_count):
|
||||
account = self.getter.get_index(i)
|
||||
self.accounts.append(account)
|
||||
logg.debug('adding account {}'.format(account))
|
||||
|
||||
return copy.copy(self.accounts)
|
||||
|
||||
|
||||
def init_legacy(config, w3):
|
||||
chain_spec = ChainSpec.from_chain_str(config.get('CIC_CHAIN_SPEC'))
|
||||
CICRegistry.init(w3, config.get('CIC_REGISTRY_ADDRESS'), chain_spec)
|
||||
CICRegistry.add_path(config.get('ETH_ABI_DIR'))
|
||||
|
||||
chain_registry = ChainRegistry(chain_spec)
|
||||
CICRegistry.add_chain_registry(chain_registry, True)
|
||||
|
||||
declarator = CICRegistry.get_contract(chain_spec, 'AddressDeclarator', interface='Declarator')
|
||||
trusted_addresses_src = config.get('CIC_TRUST_ADDRESS')
|
||||
if trusted_addresses_src == None:
|
||||
raise ValueError('At least one trusted address must be declared in CIC_TRUST_ADDRESS')
|
||||
trusted_addresses = trusted_addresses_src.split(',')
|
||||
for address in trusted_addresses:
|
||||
logg.info('using trusted address {}'.format(address))
|
||||
|
||||
oracle = DeclaratorOracleAdapter(declarator.contract, trusted_addresses)
|
||||
chain_registry.add_oracle(oracle, 'naive_erc20_oracle')
|
||||
|
||||
return CICRegistry
|
||||
@@ -1,18 +0,0 @@
|
||||
# standard imports
|
||||
import re
|
||||
|
||||
# external imports
|
||||
import web3
|
||||
|
||||
def create(url):
|
||||
# web3 input
|
||||
# TODO: Replace with chainlib
|
||||
re_websocket = r'^wss?:'
|
||||
re_http = r'^https?:'
|
||||
blockchain_provider = None
|
||||
if re.match(re_websocket, url):
|
||||
blockchain_provider = web3.Web3.WebsocketProvider(url)
|
||||
elif re.match(re_http, url):
|
||||
blockchain_provider = web3.Web3.HTTPProvider(url)
|
||||
w3 = web3.Web3(blockchain_provider)
|
||||
return w3
|
||||
@@ -1,23 +0,0 @@
|
||||
# standard imports
|
||||
import logging
|
||||
|
||||
# external imports
|
||||
from crypto_dev_signer.eth.signer import ReferenceSigner as EIP155Signer
|
||||
from crypto_dev_signer.keystore import DictKeystore
|
||||
|
||||
logg = logging.getLogger(__name__)
|
||||
|
||||
keystore = DictKeystore()
|
||||
|
||||
def from_keystore(keyfile):
|
||||
global keystore
|
||||
|
||||
# signer
|
||||
if keyfile == None:
|
||||
raise ValueError('please specify signer keystore file')
|
||||
|
||||
logg.debug('loading keystore file {}'.format(keyfile))
|
||||
address = keystore.import_keystore_file(keyfile)
|
||||
|
||||
signer = EIP155Signer(keystore)
|
||||
return (address, signer,)
|
||||
@@ -1,37 +0,0 @@
|
||||
# standard imports
|
||||
import logging
|
||||
|
||||
# external imports
|
||||
from cic_eth.api.api_task import Api
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
queue = 'cic-eth'
|
||||
name = 'account'
|
||||
|
||||
|
||||
def do(token_pair, sender, recipient, sender_balance, aux, block_number, tx_index):
|
||||
"""Triggers creation and registration of new account through the custodial cic-eth component.
|
||||
|
||||
It expects the following aux parameters to exist:
|
||||
- redis_host_callback: Redis host name exposed to cic-eth, for callback
|
||||
- redis_port_callback: Redis port exposed to cic-eth, for callback
|
||||
- redis_db: Redis db, for callback
|
||||
- redis_channel: Redis channel, for callback
|
||||
- chain_spec: Chain specification for the chain to execute the transfer on
|
||||
|
||||
See local.noop.do for details on parameters and return values.
|
||||
"""
|
||||
logg.debug('running {} {} {}'.format(__name__, token_pair, sender, recipient))
|
||||
api = Api(
|
||||
str(aux['chain_spec']),
|
||||
queue=queue,
|
||||
callback_param='{}:{}:{}:{}'.format(aux['redis_host_callback'], aux['redis_port_callback'], aux['redis_db'], aux['redis_channel']),
|
||||
callback_task='cic_eth.callbacks.redis.redis',
|
||||
callback_queue=queue,
|
||||
)
|
||||
|
||||
t = api.create_account(register=True)
|
||||
|
||||
return (None, t, sender_balance, )
|
||||
@@ -1,37 +0,0 @@
|
||||
# standard imports
|
||||
import logging
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
|
||||
def do(token_pair, sender, recipient, sender_balance, aux, block_number, tx_index):
|
||||
"""Defines the function signature for a traffic generator. The method itself only logs the input parameters.
|
||||
|
||||
If the error position in the return tuple is not None, the calling code should consider the generation as failed, and not count it towards the limit of simultaneous traffic items that can be simultaneously in flight.
|
||||
|
||||
If the task_id position in the return tuple is None, the calling code should interpret the traffic item to have been synchronously completed, and not count it towards the limit of simultaneous traffic items that can be simultaneously in flight.
|
||||
|
||||
The balance element of the result is the balance dict passed as argument, with fields updated according to the expected delta as a result of the operation. However, in the event that the generator module dispatches an asynchronous event then there is no guarantee that this balance will actually be the correct result. The caller should take care to periodically re-sync balance from the upstream.
|
||||
|
||||
:param token_pair: Source and destination tokens for the traffic item.
|
||||
:type token_pair: 2-element tuple with cic_registry.token.Token
|
||||
:param sender: Sender address
|
||||
:type sender: str, 0x-hex
|
||||
:param recipient: Recipient address
|
||||
:type recipient: str, 0x-hex
|
||||
:param sender_balance: Sender balance in full decimal resolution
|
||||
:type sender_balance: complex balance dict
|
||||
:param aux: Custom parameters defined by traffic generation client code
|
||||
:type aux: dict
|
||||
:param block_number: Syncer block number position at time of method call
|
||||
:type block_number: number
|
||||
:param tx_index: Syncer block transaction index position at time of method call
|
||||
:type tx_index: number
|
||||
:raises KeyError: Missing required aux element
|
||||
:returns: Exception|None, task_id|None and adjusted_sender_balance respectively
|
||||
:rtype: tuple
|
||||
"""
|
||||
logg.debug('running {} {} {} {} {} {} {} {}'.format(__name__, token_pair, sender, recipient, sender_balance, aux, block_number, tx_index))
|
||||
|
||||
return (None, None, sender_balance, )
|
||||
@@ -1,51 +0,0 @@
|
||||
# standard imports
|
||||
import logging
|
||||
import random
|
||||
|
||||
# external imports
|
||||
from cic_eth.api.api_task import Api
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
queue = 'cic-eth'
|
||||
name = 'erc20_transfer'
|
||||
|
||||
|
||||
def do(token_pair, sender, recipient, sender_balance, aux, block_number, tx_index):
|
||||
"""Triggers an ERC20 token transfer through the custodial cic-eth component, with a randomly chosen amount in integer resolution.
|
||||
|
||||
It expects the following aux parameters to exist:
|
||||
- redis_host_callback: Redis host name exposed to cic-eth, for callback
|
||||
- redis_port_callback: Redis port exposed to cic-eth, for callback
|
||||
- redis_db: Redis db, for callback
|
||||
- redis_channel: Redis channel, for callback
|
||||
- chain_spec: Chain specification for the chain to execute the transfer on
|
||||
|
||||
See local.noop.do for details on parameters and return values.
|
||||
"""
|
||||
logg.debug('running {} {} {} {}'.format(__name__, token_pair, sender, recipient))
|
||||
|
||||
decimals = token_pair[0].decimals()
|
||||
|
||||
sender_balance_value = sender_balance['balance_network'] - sender_balance['balance_outgoing']
|
||||
|
||||
balance_units = int(sender_balance_value / decimals)
|
||||
|
||||
if balance_units <= 0:
|
||||
return (AttributeError('sender {} has zero balance'), None, 0,)
|
||||
|
||||
spend_units = random.randint(1, balance_units)
|
||||
spend_value = spend_units * decimals
|
||||
|
||||
api = Api(
|
||||
str(aux['chain_spec']),
|
||||
queue=queue,
|
||||
callback_param='{}:{}:{}:{}'.format(aux['redis_host_callback'], aux['redis_port_callback'], aux['redis_db'], aux['redis_channel']),
|
||||
callback_task='cic_eth.callbacks.redis.redis',
|
||||
callback_queue=queue,
|
||||
)
|
||||
t = api.transfer(sender, recipient, spend_value, token_pair[0].symbol())
|
||||
|
||||
sender_balance['balance_outgoing'] += spend_value
|
||||
return (None, t, sender_balance,)
|
||||
@@ -1,101 +0,0 @@
|
||||
# standard imports
|
||||
import os
|
||||
import logging
|
||||
import re
|
||||
import sys
|
||||
import json
|
||||
|
||||
# external imports
|
||||
import redis
|
||||
import celery
|
||||
from chainsyncer.backend import MemBackend
|
||||
from chainsyncer.driver import HeadSyncer
|
||||
from chainlib.eth.connection import HTTPConnection
|
||||
from chainlib.eth.gas import DefaultGasOracle
|
||||
from chainlib.eth.nonce import DefaultNonceOracle
|
||||
from chainlib.eth.block import block_latest
|
||||
from hexathon import strip_0x
|
||||
|
||||
# local imports
|
||||
import common
|
||||
from cmd.traffic import (
|
||||
TrafficItem,
|
||||
TrafficRouter,
|
||||
TrafficProvisioner,
|
||||
TrafficSyncHandler,
|
||||
)
|
||||
from cmd.traffic import add_args as add_traffic_args
|
||||
|
||||
|
||||
# common basics
|
||||
script_dir = os.path.realpath(os.path.dirname(__file__))
|
||||
logg = common.log.create()
|
||||
argparser = common.argparse.create(script_dir, common.argparse.full_template)
|
||||
argparser = common.argparse.add(argparser, add_traffic_args, 'traffic')
|
||||
args = common.argparse.parse(argparser, logg)
|
||||
config = common.config.create(args.c, args, args.env_prefix)
|
||||
|
||||
# map custom args to local config entries
|
||||
batchsize = args.batch_size
|
||||
if batchsize < 1:
|
||||
batchsize = 1
|
||||
logg.info('batch size {}'.format(batchsize))
|
||||
config.add(batchsize, '_BATCH_SIZE', True)
|
||||
|
||||
config.add(args.redis_host_callback, '_REDIS_HOST_CALLBACK', True)
|
||||
config.add(args.redis_port_callback, '_REDIS_PORT_CALLBACK', True)
|
||||
|
||||
config.add(args.y, '_KEYSTORE_FILE', True)
|
||||
|
||||
config.add(args.q, '_CELERY_QUEUE', True)
|
||||
|
||||
common.config.log(config)
|
||||
|
||||
|
||||
def main():
|
||||
# create signer (not currently in use, but needs to be accessible for custom traffic item generators)
|
||||
(signer_address, signer) = common.signer.from_keystore(config.get('_KEYSTORE_FILE'))
|
||||
|
||||
# connect to celery
|
||||
celery.Celery(broker=config.get('CELERY_BROKER_URL'), backend=config.get('CELERY_RESULT_URL'))
|
||||
|
||||
# set up registry
|
||||
w3 = common.rpc.create(config.get('ETH_PROVIDER')) # replace with HTTPConnection when registry has been so refactored
|
||||
registry = common.registry.init_legacy(config, w3)
|
||||
|
||||
# Connect to blockchain with chainlib
|
||||
conn = HTTPConnection(config.get('ETH_PROVIDER'))
|
||||
gas_oracle = DefaultGasOracle(conn)
|
||||
nonce_oracle = DefaultNonceOracle(signer_address, conn)
|
||||
|
||||
# Set up magic traffic handler
|
||||
traffic_router = TrafficRouter()
|
||||
traffic_router.apply_import_dict(config.all(), config)
|
||||
handler = TrafficSyncHandler(config, traffic_router)
|
||||
|
||||
# Set up syncer
|
||||
syncer_backend = MemBackend(config.get('CIC_CHAIN_SPEC'), 0)
|
||||
o = block_latest()
|
||||
r = conn.do(o)
|
||||
block_offset = int(strip_0x(r), 16) + 1
|
||||
syncer_backend.set(block_offset, 0)
|
||||
|
||||
# Set up provisioner for common task input data
|
||||
TrafficProvisioner.oracles['token']= common.registry.TokenOracle(w3, config.get('CIC_CHAIN_SPEC'), registry)
|
||||
TrafficProvisioner.oracles['account'] = common.registry.AccountsOracle(w3, config.get('CIC_CHAIN_SPEC'), registry)
|
||||
TrafficProvisioner.default_aux = {
|
||||
'chain_spec': config.get('CIC_CHAIN_SPEC'),
|
||||
'registry': registry,
|
||||
'redis_host_callback': config.get('_REDIS_HOST_CALLBACK'),
|
||||
'redis_port_callback': config.get('_REDIS_PORT_CALLBACK'),
|
||||
'redis_db': config.get('REDIS_DB'),
|
||||
'api_queue': config.get('_CELERY_QUEUE'),
|
||||
}
|
||||
|
||||
syncer = HeadSyncer(syncer_backend, loop_callback=handler.refresh)
|
||||
syncer.add_filter(handler)
|
||||
syncer.loop(1, conn)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,122 +0,0 @@
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const http = require('http');
|
||||
|
||||
const cic = require('cic-client-meta');
|
||||
const crdt = require('crdt-meta');
|
||||
|
||||
//const conf = JSON.parse(fs.readFileSync('./cic.conf'));
|
||||
|
||||
const config = new crdt.Config('./config');
|
||||
config.process();
|
||||
console.log(config);
|
||||
|
||||
|
||||
function sendit(uid, envelope) {
|
||||
const d = envelope.toJSON();
|
||||
|
||||
const contentLength = (new TextEncoder().encode(d)).length;
|
||||
const opts = {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Content-Length': contentLength,
|
||||
'X-CIC-AUTOMERGE': 'client',
|
||||
|
||||
},
|
||||
};
|
||||
let url = config.get('META_URL');
|
||||
url = url.replace(new RegExp('^(.+://[^/]+)/*$'), '$1/');
|
||||
console.log('posting to url: ' + url + uid);
|
||||
const req = http.request(url + uid, opts, (res) => {
|
||||
res.on('data', process.stdout.write);
|
||||
res.on('end', () => {
|
||||
console.log('result', res.statusCode, res.headers);
|
||||
});
|
||||
});
|
||||
if (!req.write(d)) {
|
||||
console.error('foo', d);
|
||||
process.exit(1);
|
||||
}
|
||||
req.end();
|
||||
}
|
||||
|
||||
function doOne(keystore, filePath) {
|
||||
const signer = new crdt.PGPSigner(keystore);
|
||||
const parts = path.basename(filePath).split('.');
|
||||
const ethereum_address = path.basename(parts[0]);
|
||||
|
||||
cic.User.toKey('0x' + ethereum_address).then((uid) => {
|
||||
const d = fs.readFileSync(filePath, 'utf-8');
|
||||
const o = JSON.parse(d);
|
||||
//console.log(o);
|
||||
fs.unlinkSync(filePath);
|
||||
|
||||
const s = new crdt.Syncable(uid, o);
|
||||
s.setSigner(signer);
|
||||
s.onwrap = (env) => {
|
||||
sendit(uid, env);
|
||||
};
|
||||
s.sign();
|
||||
});
|
||||
}
|
||||
|
||||
const privateKeyPath = path.join(config.get('PGP_EXPORTS_DIR'), config.get('PGP_PRIVATE_KEY_FILE'));
|
||||
const publicKeyPath = path.join(config.get('PGP_EXPORTS_DIR'), config.get('PGP_PRIVATE_KEY_FILE'));
|
||||
pk = fs.readFileSync(privateKeyPath);
|
||||
pubk = fs.readFileSync(publicKeyPath);
|
||||
|
||||
new crdt.PGPKeyStore(
|
||||
config.get('PGP_PASSPHRASE'),
|
||||
pk,
|
||||
pubk,
|
||||
undefined,
|
||||
undefined,
|
||||
importMeta,
|
||||
);
|
||||
|
||||
const batchSize = 16;
|
||||
const batchDelay = 1000;
|
||||
const total = parseInt(process.argv[3]);
|
||||
const workDir = path.join(process.argv[2], 'meta');
|
||||
let count = 0;
|
||||
let batchCount = 0;
|
||||
|
||||
|
||||
function importMeta(keystore) {
|
||||
let err;
|
||||
let files;
|
||||
|
||||
try {
|
||||
err, files = fs.readdirSync(workDir);
|
||||
} catch {
|
||||
console.error('source directory not yet ready', workDir);
|
||||
setTimeout(importMeta, batchDelay, keystore);
|
||||
return;
|
||||
}
|
||||
let limit = batchSize;
|
||||
if (files.length < limit) {
|
||||
limit = files.length;
|
||||
}
|
||||
for (let i = 0; i < limit; i++) {
|
||||
const file = files[i];
|
||||
if (file.substr(-5) != '.json') {
|
||||
console.debug('skipping file', file);
|
||||
continue;
|
||||
}
|
||||
const filePath = path.join(workDir, file);
|
||||
doOne(keystore, filePath);
|
||||
count++;
|
||||
batchCount++;
|
||||
if (batchCount == batchSize) {
|
||||
console.debug('reached batch size, breathing');
|
||||
batchCount=0;
|
||||
setTimeout(importMeta, batchDelay, keystore);
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (count == total) {
|
||||
return;
|
||||
}
|
||||
setTimeout(importMeta, 100, keystore);
|
||||
}
|
||||
@@ -1,139 +0,0 @@
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const http = require('http');
|
||||
|
||||
const cic = require('cic-client-meta');
|
||||
const vcfp = require('vcard-parser');
|
||||
|
||||
//const conf = JSON.parse(fs.readFileSync('./cic.conf'));
|
||||
|
||||
const config = new cic.Config('./config');
|
||||
config.process();
|
||||
console.log(config);
|
||||
|
||||
|
||||
function sendit(uid, envelope) {
|
||||
const d = envelope.toJSON();
|
||||
|
||||
const contentLength = (new TextEncoder().encode(d)).length;
|
||||
const opts = {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Content-Length': contentLength,
|
||||
'X-CIC-AUTOMERGE': 'client',
|
||||
|
||||
},
|
||||
};
|
||||
let url = config.get('META_URL');
|
||||
url = url.replace(new RegExp('^(.+://[^/]+)/*$'), '$1/');
|
||||
console.log('posting to url: ' + url + uid);
|
||||
const req = http.request(url + uid, opts, (res) => {
|
||||
res.on('data', process.stdout.write);
|
||||
res.on('end', () => {
|
||||
console.log('result', res.statusCode, res.headers);
|
||||
});
|
||||
});
|
||||
if (!req.write(d)) {
|
||||
console.error('foo', d);
|
||||
process.exit(1);
|
||||
}
|
||||
req.end();
|
||||
}
|
||||
|
||||
function doOne(keystore, filePath, identifier) {
|
||||
const signer = new cic.PGPSigner(keystore);
|
||||
|
||||
const o = JSON.parse(fs.readFileSync(filePath).toString());
|
||||
//const b = Buffer.from(j['vcard'], 'base64');
|
||||
//const s = b.toString();
|
||||
//const o = vcfp.parse(s);
|
||||
//const phone = o.tel[0].value;
|
||||
|
||||
//cic.Phone.toKey(phone).then((uid) => {
|
||||
//const o = fs.readFileSync(filePath, 'utf-8');
|
||||
|
||||
const s = new cic.Syncable(identifier, o);
|
||||
s.setSigner(signer);
|
||||
s.onwrap = (env) => {
|
||||
sendit(identifier, env);
|
||||
};
|
||||
s.sign();
|
||||
//});
|
||||
}
|
||||
|
||||
const privateKeyPath = path.join(config.get('PGP_EXPORTS_DIR'), config.get('PGP_PRIVATE_KEY_FILE'));
|
||||
const publicKeyPath = path.join(config.get('PGP_EXPORTS_DIR'), config.get('PGP_PRIVATE_KEY_FILE'));
|
||||
pk = fs.readFileSync(privateKeyPath);
|
||||
pubk = fs.readFileSync(publicKeyPath);
|
||||
|
||||
new cic.PGPKeyStore(
|
||||
config.get('PGP_PASSPHRASE'),
|
||||
pk,
|
||||
pubk,
|
||||
undefined,
|
||||
undefined,
|
||||
importMetaCustom,
|
||||
);
|
||||
|
||||
const batchSize = 16;
|
||||
const batchDelay = 1000;
|
||||
const total = parseInt(process.argv[3]);
|
||||
const dataDir = process.argv[2];
|
||||
const workDir = path.join(dataDir, 'custom/meta');
|
||||
const userDir = path.join(dataDir, 'custom/new');
|
||||
let count = 0;
|
||||
let batchCount = 0;
|
||||
|
||||
|
||||
function importMetaCustom(keystore) {
|
||||
let err;
|
||||
let files;
|
||||
|
||||
try {
|
||||
err, files = fs.readdirSync(workDir);
|
||||
} catch {
|
||||
console.error('source directory not yet ready', workDir);
|
||||
setTimeout(importMetaPhone, batchDelay, keystore);
|
||||
return;
|
||||
}
|
||||
let limit = batchSize;
|
||||
if (files.length < limit) {
|
||||
limit = files.length;
|
||||
}
|
||||
for (let i = 0; i < limit; i++) {
|
||||
const file = files[i];
|
||||
if (file.length < 3) {
|
||||
console.debug('skipping file', file);
|
||||
continue;
|
||||
}
|
||||
//const identifier = file.substr(0,file.length-5);
|
||||
const identifier = file;
|
||||
const filePath = path.join(workDir, file);
|
||||
console.log(filePath);
|
||||
|
||||
//const address = fs.readFileSync(filePath).toString().substring(2).toUpperCase();
|
||||
const custom = JSON.parse(fs.readFileSync(filePath).toString());
|
||||
const customFilePath = path.join(
|
||||
userDir,
|
||||
identifier.substring(0, 2),
|
||||
identifier.substring(2, 4),
|
||||
identifier + '.json',
|
||||
);
|
||||
|
||||
doOne(keystore, filePath, identifier);
|
||||
fs.unlinkSync(filePath);
|
||||
count++;
|
||||
batchCount++;
|
||||
if (batchCount == batchSize) {
|
||||
console.debug('reached batch size, breathing');
|
||||
batchCount=0;
|
||||
setTimeout(importMeta, batchDelay, keystore);
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (count == total) {
|
||||
return;
|
||||
}
|
||||
setTimeout(importMetaCustom, 100, keystore);
|
||||
}
|
||||
@@ -1,134 +0,0 @@
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const http = require('http');
|
||||
|
||||
const cic = require('cic-client-meta');
|
||||
const vcfp = require('vcard-parser');
|
||||
const crdt = require('crdt-meta');
|
||||
|
||||
//const conf = JSON.parse(fs.readFileSync('./cic.conf'));
|
||||
|
||||
const config = new crdt.Config('./config');
|
||||
config.process();
|
||||
console.log(config);
|
||||
|
||||
|
||||
function sendit(uid, envelope) {
|
||||
const d = envelope.toJSON();
|
||||
|
||||
const contentLength = (new TextEncoder().encode(d)).length;
|
||||
const opts = {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'Content-Length': contentLength,
|
||||
'X-CIC-AUTOMERGE': 'client',
|
||||
|
||||
},
|
||||
};
|
||||
let url = config.get('META_URL');
|
||||
url = url.replace(new RegExp('^(.+://[^/]+)/*$'), '$1/');
|
||||
console.log('posting to url: ' + url + uid);
|
||||
const req = http.request(url + uid, opts, (res) => {
|
||||
res.on('data', process.stdout.write);
|
||||
res.on('end', () => {
|
||||
console.log('result', res.statusCode, res.headers);
|
||||
});
|
||||
});
|
||||
if (!req.write(d)) {
|
||||
console.error('foo', d);
|
||||
process.exit(1);
|
||||
}
|
||||
req.end();
|
||||
}
|
||||
|
||||
function doOne(keystore, filePath, address) {
|
||||
const signer = new crdt.PGPSigner(keystore);
|
||||
|
||||
const j = JSON.parse(fs.readFileSync(filePath).toString());
|
||||
const b = Buffer.from(j['vcard'], 'base64');
|
||||
const s = b.toString();
|
||||
const o = vcfp.parse(s);
|
||||
const phone = o.tel[0].value;
|
||||
|
||||
cic.Phone.toKey(phone).then((uid) => {
|
||||
|
||||
const s = new crdt.Syncable(uid, address);
|
||||
s.setSigner(signer);
|
||||
s.onwrap = (env) => {
|
||||
sendit(uid, env);
|
||||
};
|
||||
s.sign();
|
||||
});
|
||||
}
|
||||
|
||||
const privateKeyPath = path.join(config.get('PGP_EXPORTS_DIR'), config.get('PGP_PRIVATE_KEY_FILE'));
|
||||
const publicKeyPath = path.join(config.get('PGP_EXPORTS_DIR'), config.get('PGP_PRIVATE_KEY_FILE'));
|
||||
pk = fs.readFileSync(privateKeyPath);
|
||||
pubk = fs.readFileSync(publicKeyPath);
|
||||
|
||||
new crdt.PGPKeyStore(
|
||||
config.get('PGP_PASSPHRASE'),
|
||||
pk,
|
||||
pubk,
|
||||
undefined,
|
||||
undefined,
|
||||
importMetaPhone,
|
||||
);
|
||||
|
||||
const batchSize = 16;
|
||||
const batchDelay = 1000;
|
||||
const total = parseInt(process.argv[3]);
|
||||
const dataDir = process.argv[2];
|
||||
const workDir = path.join(dataDir, 'phone/meta');
|
||||
const userDir = path.join(dataDir, 'new');
|
||||
let count = 0;
|
||||
let batchCount = 0;
|
||||
|
||||
|
||||
function importMetaPhone(keystore) {
|
||||
let err;
|
||||
let files;
|
||||
|
||||
try {
|
||||
err, files = fs.readdirSync(workDir);
|
||||
} catch {
|
||||
console.error('source directory not yet ready', workDir);
|
||||
setTimeout(importMetaPhone, batchDelay, keystore);
|
||||
return;
|
||||
}
|
||||
let limit = batchSize;
|
||||
if (files.length < limit) {
|
||||
limit = files.length;
|
||||
}
|
||||
for (let i = 0; i < limit; i++) {
|
||||
const file = files[i];
|
||||
if (file.substr(0) == '.') {
|
||||
console.debug('skipping file', file);
|
||||
}
|
||||
const filePath = path.join(workDir, file);
|
||||
|
||||
const address = fs.readFileSync(filePath).toString().substring(2).toUpperCase();
|
||||
const metaFilePath = path.join(
|
||||
userDir,
|
||||
address.substring(0, 2),
|
||||
address.substring(2, 4),
|
||||
address + '.json',
|
||||
);
|
||||
|
||||
doOne(keystore, metaFilePath, address);
|
||||
fs.unlinkSync(filePath);
|
||||
count++;
|
||||
batchCount++;
|
||||
if (batchCount == batchSize) {
|
||||
console.debug('reached batch size, breathing');
|
||||
batchCount=0;
|
||||
setTimeout(importMetaPhone, batchDelay, keystore);
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (count == total) {
|
||||
return;
|
||||
}
|
||||
setTimeout(importMetaPhone, 100, keystore);
|
||||
}
|
||||
@@ -1,157 +0,0 @@
|
||||
# standard imports
|
||||
import os
|
||||
import sys
|
||||
import logging
|
||||
import argparse
|
||||
import hashlib
|
||||
import redis
|
||||
import celery
|
||||
|
||||
# external imports
|
||||
import confini
|
||||
from chainlib.eth.connection import EthHTTPConnection
|
||||
from chainlib.chain import ChainSpec
|
||||
from hexathon import (
|
||||
strip_0x,
|
||||
add_0x,
|
||||
)
|
||||
from chainlib.eth.address import to_checksum_address
|
||||
from crypto_dev_signer.eth.signer import ReferenceSigner as EIP155Signer
|
||||
from crypto_dev_signer.keystore.dict import DictKeystore
|
||||
from cic_types.models.person import Person
|
||||
|
||||
# local imports
|
||||
from import_util import BalanceProcessor
|
||||
from import_task import *
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
config_dir = './config'
|
||||
|
||||
argparser = argparse.ArgumentParser(description='daemon that monitors transactions in new blocks')
|
||||
argparser.add_argument('-p', '--provider', dest='p', type=str, help='chain rpc provider address')
|
||||
argparser.add_argument('-y', '--key-file', dest='y', type=str, help='Ethereum keystore file to use for signing')
|
||||
argparser.add_argument('-c', type=str, default=config_dir, help='config root to use')
|
||||
argparser.add_argument('--old-chain-spec', type=str, dest='old_chain_spec', default='evm:oldchain:1', help='chain spec')
|
||||
argparser.add_argument('-i', '--chain-spec', type=str, dest='i', help='chain spec')
|
||||
argparser.add_argument('-r', '--registry-address', type=str, dest='r', help='CIC Registry address')
|
||||
argparser.add_argument('--meta-host', dest='meta_host', type=str, help='metadata server host')
|
||||
argparser.add_argument('--meta-port', dest='meta_port', type=int, help='metadata server host')
|
||||
argparser.add_argument('--redis-host', dest='redis_host', type=str, help='redis host to use for task submission')
|
||||
argparser.add_argument('--redis-port', dest='redis_port', type=int, help='redis host to use for task submission')
|
||||
argparser.add_argument('--redis-db', dest='redis_db', type=int, help='redis db to use for task submission and callback')
|
||||
argparser.add_argument('--token-symbol', default='SRF', type=str, dest='token_symbol', help='Token symbol to use for trnsactions')
|
||||
argparser.add_argument('--head', action='store_true', help='start at current block height (overrides --offset)')
|
||||
argparser.add_argument('--env-prefix', default=os.environ.get('CONFINI_ENV_PREFIX'), dest='env_prefix', type=str, help='environment prefix for variables to overwrite configuration')
|
||||
argparser.add_argument('-q', type=str, default='cic-eth', help='celery queue to submit transaction tasks to')
|
||||
argparser.add_argument('--offset', type=int, default=0, help='block offset to start syncer from')
|
||||
argparser.add_argument('-v', help='be verbose', action='store_true')
|
||||
argparser.add_argument('-vv', help='be more verbose', action='store_true')
|
||||
argparser.add_argument('user_dir', default='out', type=str, help='user export directory')
|
||||
args = argparser.parse_args(sys.argv[1:])
|
||||
|
||||
if args.v == True:
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
elif args.vv == True:
|
||||
logging.getLogger().setLevel(logging.DEBUG)
|
||||
|
||||
config_dir = os.path.join(args.c)
|
||||
os.makedirs(config_dir, 0o777, True)
|
||||
config = confini.Config(config_dir, args.env_prefix)
|
||||
config.process()
|
||||
# override args
|
||||
args_override = {
|
||||
'CIC_CHAIN_SPEC': getattr(args, 'i'),
|
||||
'ETH_PROVIDER': getattr(args, 'p'),
|
||||
'CIC_REGISTRY_ADDRESS': getattr(args, 'r'),
|
||||
'REDIS_HOST': getattr(args, 'redis_host'),
|
||||
'REDIS_PORT': getattr(args, 'redis_port'),
|
||||
'REDIS_DB': getattr(args, 'redis_db'),
|
||||
'META_HOST': getattr(args, 'meta_host'),
|
||||
'META_PORT': getattr(args, 'meta_port'),
|
||||
}
|
||||
config.dict_override(args_override, 'cli flag')
|
||||
config.censor('PASSWORD', 'DATABASE')
|
||||
config.censor('PASSWORD', 'SSL')
|
||||
logg.debug('config loaded from {}:\n{}'.format(config_dir, config))
|
||||
|
||||
redis_host = config.get('REDIS_HOST')
|
||||
redis_port = config.get('REDIS_PORT')
|
||||
redis_db = config.get('REDIS_DB')
|
||||
r = redis.Redis(redis_host, redis_port, redis_db)
|
||||
celery_app = celery.Celery(backend=config.get('CELERY_RESULT_URL'), broker=config.get('CELERY_BROKER_URL'))
|
||||
|
||||
signer_address = None
|
||||
keystore = DictKeystore()
|
||||
if args.y != None:
|
||||
logg.debug('loading keystore file {}'.format(args.y))
|
||||
signer_address = keystore.import_keystore_file(args.y)
|
||||
logg.debug('now have key for signer address {}'.format(signer_address))
|
||||
signer = EIP155Signer(keystore)
|
||||
|
||||
queue = args.q
|
||||
chain_str = config.get('CIC_CHAIN_SPEC')
|
||||
block_offset = 0
|
||||
if args.head:
|
||||
block_offset = -1
|
||||
else:
|
||||
block_offset = args.offset
|
||||
|
||||
chain_spec = ChainSpec.from_chain_str(chain_str)
|
||||
old_chain_spec_str = args.old_chain_spec
|
||||
old_chain_spec = ChainSpec.from_chain_str(old_chain_spec_str)
|
||||
|
||||
user_dir = args.user_dir # user_out_dir from import_users.py
|
||||
|
||||
token_symbol = args.token_symbol
|
||||
|
||||
MetadataTask.meta_host = config.get('META_HOST')
|
||||
MetadataTask.meta_port = config.get('META_PORT')
|
||||
ImportTask.chain_spec = chain_spec
|
||||
|
||||
def main():
|
||||
conn = EthHTTPConnection(config.get('ETH_PROVIDER'))
|
||||
|
||||
ImportTask.balance_processor = BalanceProcessor(conn, chain_spec, config.get('CIC_REGISTRY_ADDRESS'), signer_address, signer)
|
||||
ImportTask.balance_processor.init()
|
||||
|
||||
# TODO get decimals from token
|
||||
balances = {}
|
||||
f = open('{}/balances.csv'.format(user_dir, 'r'))
|
||||
remove_zeros = 10**6
|
||||
i = 0
|
||||
while True:
|
||||
l = f.readline()
|
||||
if l == None:
|
||||
break
|
||||
r = l.split(',')
|
||||
try:
|
||||
address = to_checksum_address(r[0])
|
||||
sys.stdout.write('loading balance {} {} {}'.format(i, address, r[1]).ljust(200) + "\r")
|
||||
except ValueError:
|
||||
break
|
||||
balance = int(int(r[1].rstrip()) / remove_zeros)
|
||||
balances[address] = balance
|
||||
i += 1
|
||||
|
||||
f.close()
|
||||
|
||||
ImportTask.balances = balances
|
||||
ImportTask.count = i
|
||||
|
||||
s = celery.signature(
|
||||
'import_task.send_txs',
|
||||
[
|
||||
MetadataTask.balance_processor.nonce_offset,
|
||||
],
|
||||
queue='cic-import-ussd',
|
||||
)
|
||||
s.apply_async()
|
||||
|
||||
argv = ['worker', '-Q', 'cic-import-ussd', '--loglevel=DEBUG']
|
||||
celery_app.worker_main(argv)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,70 +0,0 @@
|
||||
# standard import
|
||||
import argparse
|
||||
import csv
|
||||
import logging
|
||||
import os
|
||||
|
||||
# third-party imports
|
||||
import celery
|
||||
import confini
|
||||
|
||||
# local imports
|
||||
from import_task import *
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
default_config_dir = './config'
|
||||
|
||||
arg_parser = argparse.ArgumentParser()
|
||||
arg_parser.add_argument('-c', type=str, default=default_config_dir, help='config root to use')
|
||||
arg_parser.add_argument('--env-prefix',
|
||||
default=os.environ.get('CONFINI_ENV_PREFIX'),
|
||||
dest='env_prefix',
|
||||
type=str,
|
||||
help='environment prefix for variables to overwrite configuration')
|
||||
arg_parser.add_argument('-q', type=str, default='cic-import-ussd', help='celery queue to submit transaction tasks to')
|
||||
arg_parser.add_argument('-v', help='be verbose', action='store_true')
|
||||
arg_parser.add_argument('-vv', help='be more verbose', action='store_true')
|
||||
arg_parser.add_argument('pins_dir', default='out', type=str, help='user export directory')
|
||||
args = arg_parser.parse_args()
|
||||
|
||||
# set log levels
|
||||
if args.v:
|
||||
logg.setLevel(logging.INFO)
|
||||
elif args.vv:
|
||||
logg.setLevel(logging.DEBUG)
|
||||
|
||||
# process configs
|
||||
config_dir = args.c
|
||||
config = confini.Config(config_dir, os.environ.get('CONFINI_ENV_PREFIX'))
|
||||
config.process()
|
||||
|
||||
celery_app = celery.Celery(broker=config.get('CELERY_BROKER_URL'), backend=config.get('CELERY_RESULT_URL'))
|
||||
|
||||
db_configs = {
|
||||
'database': config.get('DATABASE_NAME'),
|
||||
'host': config.get('DATABASE_HOST'),
|
||||
'port': config.get('DATABASE_PORT'),
|
||||
'user': config.get('DATABASE_USER'),
|
||||
'password': config.get('DATABASE_PASSWORD')
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
with open(f'{args.pins_dir}/pins.csv') as pins_file:
|
||||
phone_to_pins = [tuple(row) for row in csv.reader(pins_file)]
|
||||
|
||||
s_import_pins = celery.signature(
|
||||
'import_task.set_pins',
|
||||
(db_configs, phone_to_pins),
|
||||
queue=args.q
|
||||
)
|
||||
s_import_pins.apply_async()
|
||||
|
||||
argv = ['worker', '-Q', 'cic-import-ussd', '--loglevel=DEBUG']
|
||||
celery_app.worker_main(argv)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,276 +0,0 @@
|
||||
# standard imports
|
||||
import os
|
||||
import logging
|
||||
import urllib.parse
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
import json
|
||||
|
||||
# external imports
|
||||
import celery
|
||||
import psycopg2
|
||||
from psycopg2 import extras
|
||||
from hexathon import (
|
||||
strip_0x,
|
||||
add_0x,
|
||||
)
|
||||
from chainlib.eth.address import to_checksum_address
|
||||
from chainlib.eth.tx import (
|
||||
unpack,
|
||||
raw,
|
||||
)
|
||||
from cic_types.processor import generate_metadata_pointer
|
||||
from cic_types.models.person import Person
|
||||
|
||||
#logg = logging.getLogger().getChild(__name__)
|
||||
logg = logging.getLogger()
|
||||
|
||||
celery_app = celery.current_app
|
||||
|
||||
|
||||
class ImportTask(celery.Task):
|
||||
|
||||
balances = None
|
||||
import_dir = 'out'
|
||||
count = 0
|
||||
chain_spec = None
|
||||
balance_processor = None
|
||||
max_retries = None
|
||||
|
||||
class MetadataTask(ImportTask):
|
||||
|
||||
meta_host = None
|
||||
meta_port = None
|
||||
meta_path = ''
|
||||
meta_ssl = False
|
||||
autoretry_for = (
|
||||
urllib.error.HTTPError,
|
||||
OSError,
|
||||
)
|
||||
retry_jitter = True
|
||||
retry_backoff = True
|
||||
retry_backoff_max = 60
|
||||
|
||||
@classmethod
|
||||
def meta_url(self):
|
||||
scheme = 'http'
|
||||
if self.meta_ssl:
|
||||
scheme += 's'
|
||||
url = urllib.parse.urlparse('{}://{}:{}/{}'.format(scheme, self.meta_host, self.meta_port, self.meta_path))
|
||||
return urllib.parse.urlunparse(url)
|
||||
|
||||
|
||||
def old_address_from_phone(base_path, phone):
|
||||
pidx = generate_metadata_pointer(phone.encode('utf-8'), ':cic.phone')
|
||||
phone_idx_path = os.path.join('{}/phone/{}/{}/{}'.format(
|
||||
base_path,
|
||||
pidx[:2],
|
||||
pidx[2:4],
|
||||
pidx,
|
||||
)
|
||||
)
|
||||
f = open(phone_idx_path, 'r')
|
||||
old_address = f.read()
|
||||
f.close()
|
||||
|
||||
return old_address
|
||||
|
||||
|
||||
@celery_app.task(bind=True, base=MetadataTask)
|
||||
def resolve_phone(self, phone):
|
||||
identifier = generate_metadata_pointer(phone.encode('utf-8'), ':cic.phone')
|
||||
url = urllib.parse.urljoin(self.meta_url(), identifier)
|
||||
logg.debug('attempt getting phone pointer at {} for phone {}'.format(url, phone))
|
||||
r = urllib.request.urlopen(url)
|
||||
address = json.load(r)
|
||||
address = address.replace('"', '')
|
||||
logg.debug('address {} for phone {}'.format(address, phone))
|
||||
|
||||
return address
|
||||
|
||||
|
||||
@celery_app.task(bind=True, base=MetadataTask)
|
||||
def generate_metadata(self, address, phone):
|
||||
old_address = old_address_from_phone(self.import_dir, phone)
|
||||
|
||||
logg.debug('address {}'.format(address))
|
||||
old_address_upper = strip_0x(old_address).upper()
|
||||
metadata_path = '{}/old/{}/{}/{}.json'.format(
|
||||
self.import_dir,
|
||||
old_address_upper[:2],
|
||||
old_address_upper[2:4],
|
||||
old_address_upper,
|
||||
)
|
||||
|
||||
f = open(metadata_path, 'r')
|
||||
o = json.load(f)
|
||||
f.close()
|
||||
|
||||
u = Person.deserialize(o)
|
||||
|
||||
if u.identities.get('evm') == None:
|
||||
u.identities['evm'] = {}
|
||||
sub_chain_str = '{}:{}'.format(self.chain_spec.common_name(), self.chain_spec.network_id())
|
||||
u.identities['evm'][sub_chain_str] = [add_0x(address)]
|
||||
|
||||
new_address_clean = strip_0x(address)
|
||||
filepath = os.path.join(
|
||||
self.import_dir,
|
||||
'new',
|
||||
new_address_clean[:2].upper(),
|
||||
new_address_clean[2:4].upper(),
|
||||
new_address_clean.upper() + '.json',
|
||||
)
|
||||
os.makedirs(os.path.dirname(filepath), exist_ok=True)
|
||||
|
||||
o = u.serialize()
|
||||
f = open(filepath, 'w')
|
||||
f.write(json.dumps(o))
|
||||
f.close()
|
||||
|
||||
meta_key = generate_metadata_pointer(bytes.fromhex(new_address_clean), ':cic.person')
|
||||
meta_filepath = os.path.join(
|
||||
self.import_dir,
|
||||
'meta',
|
||||
'{}.json'.format(new_address_clean.upper()),
|
||||
)
|
||||
os.symlink(os.path.realpath(filepath), meta_filepath)
|
||||
|
||||
logg.debug('found metadata {} for phone {}'.format(o, phone))
|
||||
|
||||
return address
|
||||
|
||||
|
||||
@celery_app.task(bind=True, base=MetadataTask)
|
||||
def opening_balance_tx(self, address, phone, serial):
|
||||
|
||||
|
||||
old_address = old_address_from_phone(self.import_dir, phone)
|
||||
|
||||
k = to_checksum_address(strip_0x(old_address))
|
||||
balance = self.balances[k]
|
||||
logg.debug('found balance {} for address {} phone {}'.format(balance, old_address, phone))
|
||||
|
||||
decimal_balance = self.balance_processor.get_decimal_amount(int(balance))
|
||||
|
||||
(tx_hash_hex, o) = self.balance_processor.get_rpc_tx(address, decimal_balance, serial)
|
||||
|
||||
tx = unpack(bytes.fromhex(strip_0x(o)), self.chain_spec)
|
||||
logg.debug('generated tx token value {} to {} tx hash {}'.format(decimal_balance, address, tx_hash_hex))
|
||||
|
||||
tx_path = os.path.join(
|
||||
self.import_dir,
|
||||
'txs',
|
||||
strip_0x(tx_hash_hex),
|
||||
)
|
||||
|
||||
f = open(tx_path, 'w')
|
||||
f.write(strip_0x(o))
|
||||
f.close()
|
||||
|
||||
tx_nonce_path = os.path.join(
|
||||
self.import_dir,
|
||||
'txs',
|
||||
'.' + str(tx['nonce']),
|
||||
)
|
||||
os.symlink(os.path.realpath(tx_path), tx_nonce_path)
|
||||
|
||||
return tx['hash']
|
||||
|
||||
|
||||
@celery_app.task(bind=True, base=ImportTask, autoretry_for=(FileNotFoundError,), max_retries=None, default_retry_delay=0.1)
|
||||
def send_txs(self, nonce):
|
||||
|
||||
if nonce == self.count + self.balance_processor.nonce_offset:
|
||||
logg.info('reached nonce {} (offset {} + count {}) exiting'.format(nonce, self.balance_processor.nonce_offset, self.count))
|
||||
return
|
||||
|
||||
|
||||
logg.debug('attempt to open symlink for nonce {}'.format(nonce))
|
||||
tx_nonce_path = os.path.join(
|
||||
self.import_dir,
|
||||
'txs',
|
||||
'.' + str(nonce),
|
||||
)
|
||||
f = open(tx_nonce_path, 'r')
|
||||
tx_signed_raw_hex = f.read()
|
||||
f.close()
|
||||
|
||||
os.unlink(tx_nonce_path)
|
||||
|
||||
o = raw(add_0x(tx_signed_raw_hex))
|
||||
tx_hash_hex = self.balance_processor.conn.do(o)
|
||||
|
||||
logg.info('sent nonce {} tx hash {}'.format(nonce, tx_hash_hex)) #tx_signed_raw_hex))
|
||||
|
||||
nonce += 1
|
||||
|
||||
queue = self.request.delivery_info.get('routing_key')
|
||||
s = celery.signature(
|
||||
'import_task.send_txs',
|
||||
[
|
||||
nonce,
|
||||
],
|
||||
queue=queue,
|
||||
)
|
||||
s.apply_async()
|
||||
|
||||
|
||||
return nonce
|
||||
|
||||
|
||||
@celery_app.task
|
||||
def set_pins(config: dict, phone_to_pins: list):
|
||||
# define db connection
|
||||
db_conn = psycopg2.connect(
|
||||
database=config.get('database'),
|
||||
host=config.get('host'),
|
||||
port=config.get('port'),
|
||||
user=config.get('user'),
|
||||
password=config.get('password')
|
||||
)
|
||||
db_cursor = db_conn.cursor()
|
||||
|
||||
# update db
|
||||
for element in phone_to_pins:
|
||||
sql = 'UPDATE account SET password_hash = %s WHERE phone_number = %s'
|
||||
db_cursor.execute(sql, (element[1], element[0]))
|
||||
logg.debug(f'Updating: {element[0]} with: {element[1]}')
|
||||
|
||||
# commit changes
|
||||
db_conn.commit()
|
||||
|
||||
# close connections
|
||||
db_cursor.close()
|
||||
db_conn.close()
|
||||
|
||||
|
||||
@celery_app.task
|
||||
def set_ussd_data(config: dict, ussd_data: dict):
|
||||
# define db connection
|
||||
db_conn = psycopg2.connect(
|
||||
database=config.get('database'),
|
||||
host=config.get('host'),
|
||||
port=config.get('port'),
|
||||
user=config.get('user'),
|
||||
password=config.get('password')
|
||||
)
|
||||
db_cursor = db_conn.cursor()
|
||||
|
||||
# process ussd_data
|
||||
account_status = 1
|
||||
if ussd_data['is_activated'] == 1:
|
||||
account_status = 2
|
||||
preferred_language = ussd_data['preferred_language']
|
||||
phone_number = ussd_data['phone']
|
||||
|
||||
sql = 'UPDATE account SET account_status = %s, preferred_language = %s WHERE phone_number = %s'
|
||||
db_cursor.execute(sql, (account_status, preferred_language, phone_number))
|
||||
|
||||
# commit changes
|
||||
db_conn.commit()
|
||||
|
||||
# close connections
|
||||
db_cursor.close()
|
||||
db_conn.close()
|
||||
|
||||
@@ -1,201 +0,0 @@
|
||||
# standard imports
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import logging
|
||||
import argparse
|
||||
import uuid
|
||||
import datetime
|
||||
import time
|
||||
import urllib.request
|
||||
from glob import glob
|
||||
|
||||
# third-party imports
|
||||
import redis
|
||||
import confini
|
||||
import celery
|
||||
from hexathon import (
|
||||
add_0x,
|
||||
strip_0x,
|
||||
)
|
||||
from chainlib.eth.address import to_checksum
|
||||
from cic_types.models.person import Person
|
||||
from cic_eth.api.api_task import Api
|
||||
from chainlib.chain import ChainSpec
|
||||
from cic_types.processor import generate_metadata_pointer
|
||||
import phonenumbers
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
default_config_dir = '/usr/local/etc/cic'
|
||||
|
||||
argparser = argparse.ArgumentParser()
|
||||
argparser.add_argument('-c', type=str, default=default_config_dir, help='config file')
|
||||
argparser.add_argument('-i', '--chain-spec', dest='i', type=str, help='Chain specification string')
|
||||
argparser.add_argument('--redis-host', dest='redis_host', type=str, help='redis host to use for task submission')
|
||||
argparser.add_argument('--redis-port', dest='redis_port', type=int, help='redis host to use for task submission')
|
||||
argparser.add_argument('--redis-db', dest='redis_db', type=int, help='redis db to use for task submission and callback')
|
||||
argparser.add_argument('--batch-size', dest='batch_size', default=100, type=int, help='burst size of sending transactions to node') # batch size should be slightly below cumulative gas limit worth, eg 80000 gas txs with 8000000 limit is a bit less than 100 batch size
|
||||
argparser.add_argument('--batch-delay', dest='batch_delay', default=3, type=int, help='seconds delay between batches')
|
||||
argparser.add_argument('--timeout', default=60.0, type=float, help='Callback timeout')
|
||||
argparser.add_argument('-q', type=str, default='cic-eth', help='Task queue')
|
||||
argparser.add_argument('-v', action='store_true', help='Be verbose')
|
||||
argparser.add_argument('-vv', action='store_true', help='Be more verbose')
|
||||
argparser.add_argument('user_dir', type=str, help='path to users export dir tree')
|
||||
args = argparser.parse_args()
|
||||
|
||||
if args.v:
|
||||
logg.setLevel(logging.INFO)
|
||||
elif args.vv:
|
||||
logg.setLevel(logging.DEBUG)
|
||||
|
||||
config_dir = args.c
|
||||
config = confini.Config(config_dir, os.environ.get('CONFINI_ENV_PREFIX'))
|
||||
config.process()
|
||||
args_override = {
|
||||
'CIC_CHAIN_SPEC': getattr(args, 'i'),
|
||||
'REDIS_HOST': getattr(args, 'redis_host'),
|
||||
'REDIS_PORT': getattr(args, 'redis_port'),
|
||||
'REDIS_DB': getattr(args, 'redis_db'),
|
||||
}
|
||||
config.dict_override(args_override, 'cli')
|
||||
celery_app = celery.Celery(broker=config.get('CELERY_BROKER_URL'), backend=config.get('CELERY_RESULT_URL'))
|
||||
|
||||
redis_host = config.get('REDIS_HOST')
|
||||
redis_port = config.get('REDIS_PORT')
|
||||
redis_db = config.get('REDIS_DB')
|
||||
r = redis.Redis(redis_host, redis_port, redis_db)
|
||||
|
||||
ps = r.pubsub()
|
||||
|
||||
user_new_dir = os.path.join(args.user_dir, 'new')
|
||||
os.makedirs(user_new_dir)
|
||||
|
||||
meta_dir = os.path.join(args.user_dir, 'meta')
|
||||
os.makedirs(meta_dir)
|
||||
|
||||
user_old_dir = os.path.join(args.user_dir, 'old')
|
||||
os.stat(user_old_dir)
|
||||
|
||||
txs_dir = os.path.join(args.user_dir, 'txs')
|
||||
os.makedirs(txs_dir)
|
||||
|
||||
chain_spec = ChainSpec.from_chain_str(config.get('CIC_CHAIN_SPEC'))
|
||||
chain_str = str(chain_spec)
|
||||
|
||||
batch_size = args.batch_size
|
||||
batch_delay = args.batch_delay
|
||||
|
||||
db_configs = {
|
||||
'database': config.get('DATABASE_NAME'),
|
||||
'host': config.get('DATABASE_HOST'),
|
||||
'port': config.get('DATABASE_PORT'),
|
||||
'user': config.get('DATABASE_USER'),
|
||||
'password': config.get('DATABASE_PASSWORD')
|
||||
}
|
||||
|
||||
|
||||
def build_ussd_request(phone, host, port, service_code, username, password, ssl=False):
|
||||
url = 'http'
|
||||
if ssl:
|
||||
url += 's'
|
||||
url += '://{}:{}'.format(host, port)
|
||||
url += '/?username={}&password={}'.format(username, password) #config.get('USSD_USER'), config.get('USSD_PASS'))
|
||||
|
||||
logg.info('ussd service url {}'.format(url))
|
||||
logg.info('ussd phone {}'.format(phone))
|
||||
|
||||
session = uuid.uuid4().hex
|
||||
data = {
|
||||
'sessionId': session,
|
||||
'serviceCode': service_code,
|
||||
'phoneNumber': phone,
|
||||
'text': service_code,
|
||||
}
|
||||
req = urllib.request.Request(url)
|
||||
data_str = json.dumps(data)
|
||||
data_bytes = data_str.encode('utf-8')
|
||||
req.add_header('Content-Type', 'application/json')
|
||||
req.data = data_bytes
|
||||
|
||||
return req
|
||||
|
||||
|
||||
def register_ussd(i, u):
|
||||
phone_object = phonenumbers.parse(u.tel)
|
||||
phone = phonenumbers.format_number(phone_object, phonenumbers.PhoneNumberFormat.E164)
|
||||
logg.debug('tel {} {}'.format(u.tel, phone))
|
||||
req = build_ussd_request(phone, 'localhost', 63315, '*483*46#', '', '')
|
||||
response = urllib.request.urlopen(req)
|
||||
response_data = response.read().decode('utf-8')
|
||||
state = response_data[:3]
|
||||
out = response_data[4:]
|
||||
logg.debug('ussd reponse: {}'.format(out))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
i = 0
|
||||
j = 0
|
||||
for x in os.walk(user_old_dir):
|
||||
for y in x[2]:
|
||||
if y[len(y)-5:] != '.json':
|
||||
continue
|
||||
# handle json containing person object
|
||||
filepath = None
|
||||
if y[:15] != '_ussd_data.json':
|
||||
filepath = os.path.join(x[0], y)
|
||||
f = open(filepath, 'r')
|
||||
try:
|
||||
o = json.load(f)
|
||||
except json.decoder.JSONDecodeError as e:
|
||||
f.close()
|
||||
logg.error('load error for {}: {}'.format(y, e))
|
||||
continue
|
||||
f.close()
|
||||
u = Person.deserialize(o)
|
||||
|
||||
new_address = register_ussd(i, u)
|
||||
|
||||
phone_object = phonenumbers.parse(u.tel)
|
||||
phone = phonenumbers.format_number(phone_object, phonenumbers.PhoneNumberFormat.E164)
|
||||
|
||||
s_phone = celery.signature(
|
||||
'import_task.resolve_phone',
|
||||
[
|
||||
phone,
|
||||
],
|
||||
queue='cic-import-ussd',
|
||||
)
|
||||
|
||||
s_meta = celery.signature(
|
||||
'import_task.generate_metadata',
|
||||
[
|
||||
phone,
|
||||
],
|
||||
queue='cic-import-ussd',
|
||||
)
|
||||
|
||||
s_balance = celery.signature(
|
||||
'import_task.opening_balance_tx',
|
||||
[
|
||||
phone,
|
||||
i,
|
||||
],
|
||||
queue='cic-import-ussd',
|
||||
)
|
||||
|
||||
s_meta.link(s_balance)
|
||||
s_phone.link(s_meta)
|
||||
# block time plus a bit of time for ussd processing
|
||||
s_phone.apply_async(countdown=7)
|
||||
|
||||
i += 1
|
||||
sys.stdout.write('imported {} {}'.format(i, u).ljust(200) + "\r")
|
||||
|
||||
j += 1
|
||||
if j == batch_size:
|
||||
time.sleep(batch_delay)
|
||||
j = 0
|
||||
|
||||
@@ -1,70 +0,0 @@
|
||||
# standard imports
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
||||
# external imports
|
||||
import celery
|
||||
from confini import Config
|
||||
|
||||
# local imports
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
default_config_dir = '/usr/local/etc/cic'
|
||||
|
||||
arg_parser = argparse.ArgumentParser()
|
||||
arg_parser.add_argument('-c', type=str, default=default_config_dir, help='config file')
|
||||
arg_parser.add_argument('-q', type=str, default='cic-eth', help='Task queue')
|
||||
arg_parser.add_argument('-v', action='store_true', help='Be verbose')
|
||||
arg_parser.add_argument('-vv', action='store_true', help='Be more verbose')
|
||||
arg_parser.add_argument('user_dir', type=str, help='path to users export dir tree')
|
||||
args = arg_parser.parse_args()
|
||||
|
||||
if args.v:
|
||||
logg.setLevel(logging.INFO)
|
||||
elif args.vv:
|
||||
logg.setLevel(logging.DEBUG)
|
||||
|
||||
config_dir = args.c
|
||||
config = Config(config_dir, os.environ.get('CONFINI_ENV_PREFIX'))
|
||||
config.process()
|
||||
|
||||
user_old_dir = os.path.join(args.user_dir, 'old')
|
||||
os.stat(user_old_dir)
|
||||
|
||||
db_configs = {
|
||||
'database': config.get('DATABASE_NAME'),
|
||||
'host': config.get('DATABASE_HOST'),
|
||||
'port': config.get('DATABASE_PORT'),
|
||||
'user': config.get('DATABASE_USER'),
|
||||
'password': config.get('DATABASE_PASSWORD')
|
||||
}
|
||||
celery_app = celery.Celery(broker=config.get('CELERY_BROKER_URL'), backend=config.get('CELERY_RESULT_URL'))
|
||||
|
||||
if __name__ == '__main__':
|
||||
for x in os.walk(user_old_dir):
|
||||
for y in x[2]:
|
||||
|
||||
if y[len(y) - 5:] != '.json':
|
||||
continue
|
||||
|
||||
# handle ussd_data json object
|
||||
if y[:15] == '_ussd_data.json':
|
||||
filepath = os.path.join(x[0], y)
|
||||
f = open(filepath, 'r')
|
||||
try:
|
||||
ussd_data = json.load(f)
|
||||
except json.decoder.JSONDecodeError as e:
|
||||
f.close()
|
||||
logg.error('load error for {}: {}'.format(y, e))
|
||||
continue
|
||||
f.close()
|
||||
|
||||
s_set_ussd_data = celery.signature(
|
||||
'import_task.set_ussd_data',
|
||||
[db_configs, ussd_data]
|
||||
)
|
||||
s_set_ussd_data.apply_async(queue='cic-import-ussd')
|
||||
@@ -1,72 +0,0 @@
|
||||
# standard imports
|
||||
import logging
|
||||
|
||||
# external imports
|
||||
from eth_contract_registry import Registry
|
||||
from eth_token_index import TokenUniqueSymbolIndex
|
||||
from chainlib.eth.gas import OverrideGasOracle
|
||||
from chainlib.eth.nonce import OverrideNonceOracle
|
||||
from chainlib.eth.erc20 import ERC20
|
||||
from chainlib.eth.tx import (
|
||||
count,
|
||||
TxFormat,
|
||||
)
|
||||
|
||||
logg = logging.getLogger().getChild(__name__)
|
||||
|
||||
|
||||
class BalanceProcessor:
|
||||
|
||||
def __init__(self, conn, chain_spec, registry_address, signer_address, signer):
|
||||
|
||||
self.chain_spec = chain_spec
|
||||
self.conn = conn
|
||||
#self.signer_address = signer_address
|
||||
self.registry_address = registry_address
|
||||
|
||||
self.token_index_address = None
|
||||
self.token_address = None
|
||||
self.signer_address = signer_address
|
||||
self.signer = signer
|
||||
|
||||
o = count(signer_address)
|
||||
c = self.conn.do(o)
|
||||
self.nonce_offset = int(c, 16)
|
||||
self.gas_oracle = OverrideGasOracle(conn=conn, limit=8000000)
|
||||
|
||||
self.value_multiplier = 1
|
||||
|
||||
|
||||
def init(self):
|
||||
# Get Token registry address
|
||||
registry = Registry(self.chain_spec)
|
||||
o = registry.address_of(self.registry_address, 'TokenRegistry')
|
||||
r = self.conn.do(o)
|
||||
self.token_index_address = registry.parse_address_of(r)
|
||||
logg.info('found token index address {}'.format(self.token_index_address))
|
||||
|
||||
token_registry = TokenUniqueSymbolIndex(self.chain_spec)
|
||||
o = token_registry.address_of(self.token_index_address, 'SRF')
|
||||
r = self.conn.do(o)
|
||||
self.token_address = token_registry.parse_address_of(r)
|
||||
logg.info('found SRF token address {}'.format(self.token_address))
|
||||
|
||||
tx_factory = ERC20(self.chain_spec)
|
||||
o = tx_factory.decimals(self.token_address)
|
||||
r = self.conn.do(o)
|
||||
n = tx_factory.parse_decimals(r)
|
||||
self.value_multiplier = 10 ** n
|
||||
|
||||
|
||||
def get_rpc_tx(self, recipient, value, i):
|
||||
logg.debug('initiating nonce offset {} for recipient {}'.format(self.nonce_offset + i, recipient))
|
||||
nonce_oracle = OverrideNonceOracle(self.signer_address, self.nonce_offset + i)
|
||||
tx_factory = ERC20(self.chain_spec, signer=self.signer, nonce_oracle=nonce_oracle, gas_oracle=self.gas_oracle)
|
||||
return tx_factory.transfer(self.token_address, self.signer_address, recipient, value, tx_format=TxFormat.RLP_SIGNED)
|
||||
#(tx_hash_hex, o) = tx_factory.transfer(self.token_address, self.signer_address, recipient, value)
|
||||
#self.conn.do(o)
|
||||
#return tx_hash_hex
|
||||
|
||||
|
||||
def get_decimal_amount(self, value):
|
||||
return value * self.value_multiplier
|
||||
@@ -1,24 +0,0 @@
|
||||
[app]
|
||||
ALLOWED_IP=0.0.0.0/0
|
||||
LOCALE_FALLBACK=en
|
||||
LOCALE_PATH=/usr/src/cic-ussd/var/lib/locale/
|
||||
MAX_BODY_LENGTH=1024
|
||||
PASSWORD_PEPPER=QYbzKff6NhiQzY3ygl2BkiKOpER8RE/Upqs/5aZWW+I=
|
||||
SERVICE_CODE=*483*46#
|
||||
|
||||
[phone_number]
|
||||
REGION=KE
|
||||
|
||||
[ussd]
|
||||
MENU_FILE=/usr/src/data/ussd_menu.json
|
||||
user =
|
||||
pass =
|
||||
|
||||
[statemachine]
|
||||
STATES=/usr/src/cic-ussd/states/
|
||||
TRANSITIONS=/usr/src/cic-ussd/transitions/
|
||||
|
||||
[client]
|
||||
host =
|
||||
port =
|
||||
ssl =
|
||||
@@ -1,3 +0,0 @@
|
||||
[celery]
|
||||
broker_url = redis://localhost:63379
|
||||
result_url = redis://localhost:63379
|
||||
@@ -1,9 +0,0 @@
|
||||
[cic]
|
||||
registry_address = 0x32E860c2A0645d1B7B005273696905F5D6DC5D05
|
||||
token_index_address =
|
||||
accounts_index_address =
|
||||
declarator_address =
|
||||
approval_escrow_address =
|
||||
chain_spec = evm:bloxberg:8996
|
||||
tx_retry_delay =
|
||||
trust_address = 0xEb3907eCad74a0013c259D5874AE7f22DcBcC95C
|
||||
@@ -1,5 +0,0 @@
|
||||
[database]
|
||||
name = sempo
|
||||
host = localhost
|
||||
port = 5432
|
||||
user = postgres
|
||||
@@ -1,8 +0,0 @@
|
||||
[eth]
|
||||
#ws_provider = ws://localhost:8546
|
||||
#ttp_provider = http://localhost:8545
|
||||
provider = http://localhost:63545
|
||||
gas_provider_address =
|
||||
#chain_id =
|
||||
abi_dir = /usr/local/share/cic/solidity/abi
|
||||
account_accounts_index_writer =
|
||||
@@ -1,5 +0,0 @@
|
||||
[meta]
|
||||
url = http://localhost:63380
|
||||
host = localhost
|
||||
port = 63380
|
||||
ssl = 0
|
||||
@@ -1,5 +0,0 @@
|
||||
[pgp]
|
||||
exports_dir = ../testdata/pgp
|
||||
private_key_file = privatekeys_meta.asc
|
||||
public_key_file = publickeys_meta.asc
|
||||
passphrase = merman
|
||||
@@ -1,4 +0,0 @@
|
||||
[redis]
|
||||
host = localhost
|
||||
port = 63379
|
||||
db = 0
|
||||
@@ -1,4 +0,0 @@
|
||||
[traffic]
|
||||
#local.noop_traffic = 2
|
||||
local.account = 2
|
||||
local.transfer = 2
|
||||
@@ -1,90 +0,0 @@
|
||||
# standard imports
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
|
||||
# third-party imports
|
||||
import bcrypt
|
||||
import celery
|
||||
import confini
|
||||
import phonenumbers
|
||||
import random
|
||||
from cic_types.models.person import Person
|
||||
from cryptography.fernet import Fernet
|
||||
|
||||
# local imports
|
||||
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
script_dir = os.path.realpath(os.path.dirname(__file__))
|
||||
default_config_dir = os.environ.get('CONFINI_DIR', os.path.join(script_dir, 'config'))
|
||||
|
||||
arg_parser = argparse.ArgumentParser()
|
||||
arg_parser.add_argument('-c', type=str, default=default_config_dir, help='Config dir')
|
||||
arg_parser.add_argument('-v', action='store_true', help='Be verbose')
|
||||
arg_parser.add_argument('-vv', action='store_true', help='Be more verbose')
|
||||
arg_parser.add_argument('--userdir', type=str, help='path to users export dir tree')
|
||||
arg_parser.add_argument('pins_dir', type=str, help='path to pin export dir tree')
|
||||
|
||||
|
||||
args = arg_parser.parse_args()
|
||||
|
||||
if args.v:
|
||||
logg.setLevel(logging.INFO)
|
||||
elif args.vv:
|
||||
logg.setLevel(logging.DEBUG)
|
||||
|
||||
config = confini.Config(args.c, os.environ.get('CONFINI_ENV_PREFIX'))
|
||||
config.process()
|
||||
logg.info('loaded config\n{}'.format(config))
|
||||
|
||||
celery_app = celery.Celery(broker=config.get('CELERY_BROKER_URL'), backend=config.get('CELERY_RESULT_URL'))
|
||||
|
||||
user_dir = args.userdir
|
||||
pins_dir = args.pins_dir
|
||||
|
||||
|
||||
def generate_password_hash():
|
||||
key = Fernet.generate_key()
|
||||
fnt = Fernet(key)
|
||||
pin = str(random.randint(1000, 9999))
|
||||
return fnt.encrypt(bcrypt.hashpw(pin.encode('utf-8'), bcrypt.gensalt())).decode()
|
||||
|
||||
|
||||
user_old_dir = os.path.join(user_dir, 'old')
|
||||
logg.debug(f'reading user data from: {user_old_dir}')
|
||||
|
||||
pins_file = open(f'{pins_dir}/pins.csv', 'w')
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
for x in os.walk(user_old_dir):
|
||||
for y in x[2]:
|
||||
# skip non-json files
|
||||
if y[len(y) - 5:] != '.json':
|
||||
continue
|
||||
|
||||
# define file path for
|
||||
filepath = None
|
||||
if y[:15] != '_ussd_data.json':
|
||||
filepath = os.path.join(x[0], y)
|
||||
f = open(filepath, 'r')
|
||||
try:
|
||||
o = json.load(f)
|
||||
except json.decoder.JSONDecodeError as e:
|
||||
f.close()
|
||||
logg.error('load error for {}: {}'.format(y, e))
|
||||
continue
|
||||
f.close()
|
||||
u = Person.deserialize(o)
|
||||
|
||||
phone_object = phonenumbers.parse(u.tel)
|
||||
phone = phonenumbers.format_number(phone_object, phonenumbers.PhoneNumberFormat.E164)
|
||||
password_hash = generate_password_hash()
|
||||
pins_file.write(f'{phone},{password_hash}\n')
|
||||
logg.info(f'Writing phone: {phone}, password_hash: {password_hash}')
|
||||
|
||||
pins_file.close()
|
||||
@@ -1,247 +0,0 @@
|
||||
#!/usr/bin/python
|
||||
|
||||
# standard imports
|
||||
import json
|
||||
import time
|
||||
import datetime
|
||||
import random
|
||||
import logging
|
||||
import os
|
||||
import base64
|
||||
import hashlib
|
||||
import sys
|
||||
import argparse
|
||||
import random
|
||||
|
||||
# external imports
|
||||
import vobject
|
||||
import celery
|
||||
from faker import Faker
|
||||
import confini
|
||||
from cic_types.models.person import (
|
||||
Person,
|
||||
generate_vcard_from_contact_data,
|
||||
get_contact_data_from_vcard,
|
||||
)
|
||||
from chainlib.eth.address import to_checksum_address
|
||||
import phonenumbers
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
fake = Faker(['sl', 'en_US', 'no', 'de', 'ro'])
|
||||
|
||||
script_dir = os.path.realpath(os.path.dirname(__file__))
|
||||
#config_dir = os.environ.get('CONFINI_DIR', '/usr/local/etc/cic')
|
||||
config_dir = os.environ.get('CONFINI_DIR', os.path.join(script_dir, 'config'))
|
||||
|
||||
argparser = argparse.ArgumentParser()
|
||||
argparser.add_argument('-c', type=str, default=config_dir, help='Config dir')
|
||||
argparser.add_argument('--tag', type=str, action='append', help='Tags to add to record')
|
||||
argparser.add_argument('--gift-threshold', type=int, help='If set, users will be funded with additional random balance (in token integer units)')
|
||||
argparser.add_argument('-v', action='store_true', help='Be verbose')
|
||||
argparser.add_argument('-vv', action='store_true', help='Be more verbose')
|
||||
argparser.add_argument('--dir', default='out', type=str, help='path to users export dir tree')
|
||||
argparser.add_argument('user_count', type=int, help='amount of users to generate')
|
||||
args = argparser.parse_args()
|
||||
|
||||
if args.v:
|
||||
logg.setLevel(logging.INFO)
|
||||
elif args.vv:
|
||||
logg.setLevel(logging.DEBUG)
|
||||
|
||||
config = confini.Config(args.c, os.environ.get('CONFINI_ENV_PREFIX'))
|
||||
config.process()
|
||||
logg.info('loaded config\n{}'.format(config))
|
||||
|
||||
|
||||
dt_now = datetime.datetime.utcnow()
|
||||
dt_then = dt_now - datetime.timedelta(weeks=150)
|
||||
ts_now = int(dt_now.timestamp())
|
||||
ts_then = int(dt_then.timestamp())
|
||||
|
||||
celery_app = celery.Celery(broker=config.get('CELERY_BROKER_URL'), backend=config.get('CELERY_RESULT_URL'))
|
||||
|
||||
gift_max = args.gift_threshold or 0
|
||||
gift_factor = (10**6)
|
||||
|
||||
categories = [
|
||||
"food/water",
|
||||
"fuel/energy",
|
||||
"education",
|
||||
"health",
|
||||
"shop",
|
||||
"environment",
|
||||
"transport",
|
||||
"farming/labor",
|
||||
"savingsgroup",
|
||||
]
|
||||
|
||||
phone_idx = []
|
||||
|
||||
user_dir = args.dir
|
||||
user_count = args.user_count
|
||||
|
||||
tags = args.tag
|
||||
if tags == None or len(tags) == 0:
|
||||
tags = ['individual']
|
||||
|
||||
random.seed()
|
||||
|
||||
def genPhoneIndex(phone):
|
||||
h = hashlib.new('sha256')
|
||||
h.update(phone.encode('utf-8'))
|
||||
h.update(b':cic.phone')
|
||||
return h.digest().hex()
|
||||
|
||||
|
||||
def genId(addr, typ):
|
||||
h = hashlib.new('sha256')
|
||||
h.update(bytes.fromhex(addr[2:]))
|
||||
h.update(typ.encode('utf-8'))
|
||||
return h.digest().hex()
|
||||
|
||||
|
||||
def genDate():
|
||||
|
||||
ts = random.randint(ts_then, ts_now)
|
||||
return int(datetime.datetime.fromtimestamp(ts).timestamp())
|
||||
|
||||
|
||||
def genPhone():
|
||||
phone_str = '+254' + str(random.randint(100000000, 999999999))
|
||||
phone_object = phonenumbers.parse(phone_str)
|
||||
return phonenumbers.format_number(phone_object, phonenumbers.PhoneNumberFormat.E164)
|
||||
|
||||
|
||||
def genPersonal(phone):
|
||||
fn = fake.first_name()
|
||||
ln = fake.last_name()
|
||||
e = fake.email()
|
||||
|
||||
return generate_vcard_from_contact_data(ln, fn, phone, e)
|
||||
|
||||
|
||||
def genCats():
|
||||
i = random.randint(0, 3)
|
||||
return random.choices(categories, k=i)
|
||||
|
||||
|
||||
def genAmount():
|
||||
return random.randint(0, gift_max) * gift_factor
|
||||
|
||||
|
||||
def genDob():
|
||||
dob_src = fake.date_of_birth(minimum_age=15)
|
||||
dob = {}
|
||||
|
||||
if random.random() < 0.5:
|
||||
dob['year'] = dob_src.year
|
||||
|
||||
if random.random() > 0.5:
|
||||
dob['month'] = dob_src.month
|
||||
dob['day'] = dob_src.day
|
||||
|
||||
return dob
|
||||
|
||||
|
||||
def gen():
|
||||
old_blockchain_address = '0x' + os.urandom(20).hex()
|
||||
old_blockchain_checksum_address = to_checksum_address(old_blockchain_address)
|
||||
gender = random.choice(['female', 'male', 'other'])
|
||||
phone = genPhone()
|
||||
city = fake.city_name()
|
||||
v = genPersonal(phone)
|
||||
|
||||
contact_data = get_contact_data_from_vcard(v)
|
||||
p = Person()
|
||||
p.load_vcard(contact_data)
|
||||
|
||||
p.date_registered = genDate()
|
||||
p.date_of_birth = genDob()
|
||||
p.gender = gender
|
||||
p.identities = {
|
||||
'evm': {
|
||||
'oldchain:1': [
|
||||
old_blockchain_checksum_address,
|
||||
],
|
||||
},
|
||||
}
|
||||
p.location['area_name'] = city
|
||||
if random.randint(0, 1):
|
||||
p.location['latitude'] = (random.random() + 180) - 90 #fake.local_latitude()
|
||||
p.location['longitude'] = (random.random() + 360) - 180 #fake.local_latitude()
|
||||
|
||||
|
||||
return (old_blockchain_checksum_address, phone, p)
|
||||
|
||||
|
||||
def prepareLocalFilePath(datadir, address):
|
||||
parts = [
|
||||
address[:2],
|
||||
address[2:4],
|
||||
]
|
||||
dirs = '{}/{}/{}'.format(
|
||||
datadir,
|
||||
parts[0],
|
||||
parts[1],
|
||||
)
|
||||
os.makedirs(dirs, exist_ok=True)
|
||||
return dirs
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
base_dir = os.path.join(user_dir, 'old')
|
||||
ussd_dir = os.path.join(user_dir, 'ussd')
|
||||
os.makedirs(base_dir, exist_ok=True)
|
||||
|
||||
fa = open(os.path.join(user_dir, 'balances.csv'), 'w')
|
||||
ft = open(os.path.join(user_dir, 'tags.csv'), 'w')
|
||||
|
||||
i = 0
|
||||
while i < user_count:
|
||||
eth = None
|
||||
phone = None
|
||||
o = None
|
||||
try:
|
||||
(eth, phone, o) = gen()
|
||||
except Exception as e:
|
||||
logg.warning('generate failed, trying anew: {}'.format(e))
|
||||
continue
|
||||
uid = eth[2:].upper()
|
||||
|
||||
print(o)
|
||||
|
||||
ussd_data = {
|
||||
'phone': phone,
|
||||
'is_activated': 1,
|
||||
'preferred_language': random.sample(['en', 'sw'], 1)[0],
|
||||
'is_disabled': False
|
||||
}
|
||||
|
||||
d = prepareLocalFilePath(base_dir, uid)
|
||||
f = open('{}/{}'.format(d, uid + '.json'), 'w')
|
||||
json.dump(o.serialize(), f)
|
||||
f.close()
|
||||
|
||||
d = prepareLocalFilePath(ussd_dir, uid)
|
||||
x = open('{}/{}'.format(d, uid + '_ussd_data.json'), 'w')
|
||||
json.dump(ussd_data, x)
|
||||
x.close()
|
||||
|
||||
pidx = genPhoneIndex(phone)
|
||||
d = prepareLocalFilePath(os.path.join(user_dir, 'phone'), pidx)
|
||||
f = open('{}/{}'.format(d, pidx), 'w')
|
||||
f.write(eth)
|
||||
f.close()
|
||||
|
||||
ft.write('{}:{}\n'.format(eth, ','.join(tags)))
|
||||
amount = genAmount()
|
||||
fa.write('{},{}\n'.format(eth,amount))
|
||||
logg.debug('pidx {}, uid {}, eth {}, amount {}, phone {}'.format(pidx, uid, eth, amount, phone))
|
||||
|
||||
i += 1
|
||||
|
||||
ft.close()
|
||||
fa.close()
|
||||
@@ -1,309 +0,0 @@
|
||||
# standard imports
|
||||
import os
|
||||
import sys
|
||||
import logging
|
||||
import time
|
||||
import argparse
|
||||
import sys
|
||||
import re
|
||||
import hashlib
|
||||
import csv
|
||||
import json
|
||||
|
||||
# external imports
|
||||
import eth_abi
|
||||
import confini
|
||||
from hexathon import (
|
||||
strip_0x,
|
||||
add_0x,
|
||||
)
|
||||
from chainsyncer.backend.memory import MemBackend
|
||||
from chainsyncer.driver import HeadSyncer
|
||||
from chainlib.eth.connection import EthHTTPConnection
|
||||
from chainlib.eth.block import (
|
||||
block_latest,
|
||||
block_by_number,
|
||||
Block,
|
||||
)
|
||||
from chainlib.hash import keccak256_string_to_hex
|
||||
from chainlib.eth.address import to_checksum_address
|
||||
from chainlib.eth.gas import OverrideGasOracle
|
||||
from chainlib.eth.nonce import RPCNonceOracle
|
||||
from chainlib.eth.tx import TxFactory
|
||||
from chainlib.jsonrpc import jsonrpc_template
|
||||
from chainlib.eth.error import EthException
|
||||
from chainlib.chain import ChainSpec
|
||||
from crypto_dev_signer.eth.signer import ReferenceSigner as EIP155Signer
|
||||
from crypto_dev_signer.keystore.dict import DictKeystore
|
||||
from cic_types.models.person import Person
|
||||
from eth_erc20 import ERC20
|
||||
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
config_dir = './config'
|
||||
|
||||
argparser = argparse.ArgumentParser(description='daemon that monitors transactions in new blocks')
|
||||
argparser.add_argument('-p', '--provider', dest='p', type=str, help='chain rpc provider address')
|
||||
argparser.add_argument('-y', '--key-file', dest='y', type=str, help='Ethereum keystore file to use for signing')
|
||||
argparser.add_argument('-c', type=str, default=config_dir, help='config root to use')
|
||||
argparser.add_argument('--old-chain-spec', type=str, dest='old_chain_spec', default='evm:oldchain:1', help='chain spec')
|
||||
argparser.add_argument('-i', '--chain-spec', type=str, dest='i', help='chain spec')
|
||||
argparser.add_argument('-r', '--registry-address', type=str, dest='r', help='CIC Registry address')
|
||||
argparser.add_argument('--token-symbol', default='SRF', type=str, dest='token_symbol', help='Token symbol to use for trnsactions')
|
||||
argparser.add_argument('--head', action='store_true', help='start at current block height (overrides --offset)')
|
||||
argparser.add_argument('--env-prefix', default=os.environ.get('CONFINI_ENV_PREFIX'), dest='env_prefix', type=str, help='environment prefix for variables to overwrite configuration')
|
||||
argparser.add_argument('-q', type=str, default='cic-eth', help='celery queue to submit transaction tasks to')
|
||||
argparser.add_argument('--offset', type=int, default=0, help='block offset to start syncer from')
|
||||
argparser.add_argument('-v', help='be verbose', action='store_true')
|
||||
argparser.add_argument('-vv', help='be more verbose', action='store_true')
|
||||
argparser.add_argument('user_dir', type=str, help='user export directory')
|
||||
args = argparser.parse_args(sys.argv[1:])
|
||||
|
||||
if args.v == True:
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
elif args.vv == True:
|
||||
logging.getLogger().setLevel(logging.DEBUG)
|
||||
|
||||
config_dir = os.path.join(args.c)
|
||||
os.makedirs(config_dir, 0o777, True)
|
||||
config = confini.Config(config_dir, args.env_prefix)
|
||||
config.process()
|
||||
# override args
|
||||
args_override = {
|
||||
'CIC_CHAIN_SPEC': getattr(args, 'i'),
|
||||
'ETH_PROVIDER': getattr(args, 'p'),
|
||||
'CIC_REGISTRY_ADDRESS': getattr(args, 'r'),
|
||||
}
|
||||
config.dict_override(args_override, 'cli flag')
|
||||
config.censor('PASSWORD', 'DATABASE')
|
||||
config.censor('PASSWORD', 'SSL')
|
||||
logg.debug('config loaded from {}:\n{}'.format(config_dir, config))
|
||||
|
||||
#app = celery.Celery(backend=config.get('CELERY_RESULT_URL'), broker=config.get('CELERY_BROKER_URL'))
|
||||
|
||||
signer_address = None
|
||||
keystore = DictKeystore()
|
||||
if args.y != None:
|
||||
logg.debug('loading keystore file {}'.format(args.y))
|
||||
signer_address = keystore.import_keystore_file(args.y)
|
||||
logg.debug('now have key for signer address {}'.format(signer_address))
|
||||
signer = EIP155Signer(keystore)
|
||||
|
||||
queue = args.q
|
||||
chain_str = config.get('CIC_CHAIN_SPEC')
|
||||
block_offset = 0
|
||||
if args.head:
|
||||
block_offset = -1
|
||||
else:
|
||||
block_offset = args.offset
|
||||
|
||||
chain_spec = ChainSpec.from_chain_str(chain_str)
|
||||
old_chain_spec_str = args.old_chain_spec
|
||||
old_chain_spec = ChainSpec.from_chain_str(old_chain_spec_str)
|
||||
|
||||
user_dir = args.user_dir # user_out_dir from import_users.py
|
||||
|
||||
token_symbol = args.token_symbol
|
||||
|
||||
|
||||
class Handler:
|
||||
|
||||
account_index_add_signature = keccak256_string_to_hex('add(address)')[:8]
|
||||
|
||||
def __init__(self, conn, chain_spec, user_dir, balances, token_address, signer, gas_oracle, nonce_oracle):
|
||||
self.token_address = token_address
|
||||
self.user_dir = user_dir
|
||||
self.balances = balances
|
||||
self.chain_spec = chain_spec
|
||||
self.tx_factory = ERC20(chain_spec, signer, gas_oracle, nonce_oracle)
|
||||
|
||||
|
||||
def name(self):
|
||||
return 'balance_handler'
|
||||
|
||||
|
||||
def filter(self, conn, block, tx, db_session):
|
||||
if tx.payload == None or len(tx.payload) == 0:
|
||||
logg.debug('no payload, skipping {}'.format(tx))
|
||||
return
|
||||
|
||||
if tx.payload[:8] == self.account_index_add_signature:
|
||||
recipient = eth_abi.decode_single('address', bytes.fromhex(tx.payload[-64:]))
|
||||
#original_address = to_checksum_address(self.addresses[to_checksum_address(recipient)])
|
||||
user_file = 'new/{}/{}/{}.json'.format(
|
||||
recipient[2:4].upper(),
|
||||
recipient[4:6].upper(),
|
||||
recipient[2:].upper(),
|
||||
)
|
||||
filepath = os.path.join(self.user_dir, user_file)
|
||||
o = None
|
||||
try:
|
||||
f = open(filepath, 'r')
|
||||
o = json.load(f)
|
||||
f.close()
|
||||
except FileNotFoundError:
|
||||
logg.error('no import record of address {}'.format(recipient))
|
||||
return
|
||||
u = Person.deserialize(o)
|
||||
original_address = u.identities[old_chain_spec.engine()]['{}:{}'.format(old_chain_spec.common_name(), old_chain_spec.network_id())][0]
|
||||
try:
|
||||
balance = self.balances[original_address]
|
||||
except KeyError as e:
|
||||
logg.error('balance get fail orig {} new {}'.format(original_address, recipient))
|
||||
return
|
||||
|
||||
# TODO: store token object in handler ,get decimals from there
|
||||
multiplier = 10**6
|
||||
balance_full = balance * multiplier
|
||||
logg.info('registered {} originally {} ({}) tx hash {} balance {}'.format(recipient, original_address, u, tx.hash, balance_full))
|
||||
|
||||
(tx_hash_hex, o) = self.tx_factory.transfer(self.token_address, signer_address, recipient, balance_full)
|
||||
logg.info('submitting erc20 transfer tx {} for recipient {}'.format(tx_hash_hex, recipient))
|
||||
r = conn.do(o)
|
||||
|
||||
tx_path = os.path.join(
|
||||
user_dir,
|
||||
'txs',
|
||||
strip_0x(tx_hash_hex),
|
||||
)
|
||||
f = open(tx_path, 'w')
|
||||
f.write(strip_0x(o['params'][0]))
|
||||
f.close()
|
||||
# except TypeError as e:
|
||||
# logg.warning('typerror {}'.format(e))
|
||||
# pass
|
||||
# except IndexError as e:
|
||||
# logg.warning('indexerror {}'.format(e))
|
||||
# pass
|
||||
# except EthException as e:
|
||||
# logg.error('send error {}'.format(e).ljust(200))
|
||||
#except KeyError as e:
|
||||
# logg.error('key record not found in imports: {}'.format(e).ljust(200))
|
||||
|
||||
|
||||
#class BlockGetter:
|
||||
#
|
||||
# def __init__(self, conn, gas_oracle, nonce_oracle, chain_spec):
|
||||
# self.conn = conn
|
||||
# self.tx_factory = ERC20(signer=signer, gas_oracle=gas_oracle, nonce_oracle=nonce_oracle, chain_id=chain_id)
|
||||
#
|
||||
#
|
||||
# def get(self, n):
|
||||
# o = block_by_number(n)
|
||||
# r = self.conn.do(o)
|
||||
# b = None
|
||||
# try:
|
||||
# b = Block(r)
|
||||
# except TypeError as e:
|
||||
# if r == None:
|
||||
# logg.debug('block not found {}'.format(n))
|
||||
# else:
|
||||
# logg.error('block retrieve error {}'.format(e))
|
||||
# return b
|
||||
|
||||
|
||||
def progress_callback(block_number, tx_index):
|
||||
sys.stdout.write(str(block_number).ljust(200) + "\n")
|
||||
|
||||
|
||||
|
||||
def main():
|
||||
global chain_str, block_offset, user_dir
|
||||
|
||||
conn = EthHTTPConnection(config.get('ETH_PROVIDER'))
|
||||
gas_oracle = OverrideGasOracle(conn=conn, limit=8000000)
|
||||
nonce_oracle = RPCNonceOracle(signer_address, conn)
|
||||
|
||||
# Get Token registry address
|
||||
txf = TxFactory(chain_spec, signer=signer, gas_oracle=gas_oracle, nonce_oracle=None)
|
||||
tx = txf.template(signer_address, config.get('CIC_REGISTRY_ADDRESS'))
|
||||
|
||||
registry_addressof_method = keccak256_string_to_hex('addressOf(bytes32)')[:8]
|
||||
data = add_0x(registry_addressof_method)
|
||||
data += eth_abi.encode_single('bytes32', b'TokenRegistry').hex()
|
||||
txf.set_code(tx, data)
|
||||
|
||||
o = jsonrpc_template()
|
||||
o['method'] = 'eth_call'
|
||||
o['params'].append(txf.normalize(tx))
|
||||
o['params'].append('latest')
|
||||
r = conn.do(o)
|
||||
token_index_address = to_checksum_address(eth_abi.decode_single('address', bytes.fromhex(strip_0x(r))))
|
||||
logg.info('found token index address {}'.format(token_index_address))
|
||||
|
||||
|
||||
# Get Sarafu token address
|
||||
tx = txf.template(signer_address, token_index_address)
|
||||
data = add_0x(registry_addressof_method)
|
||||
h = hashlib.new('sha256')
|
||||
h.update(token_symbol.encode('utf-8'))
|
||||
z = h.digest()
|
||||
data += eth_abi.encode_single('bytes32', z).hex()
|
||||
txf.set_code(tx, data)
|
||||
o = jsonrpc_template()
|
||||
o['method'] = 'eth_call'
|
||||
o['params'].append(txf.normalize(tx))
|
||||
o['params'].append('latest')
|
||||
r = conn.do(o)
|
||||
try:
|
||||
sarafu_token_address = to_checksum_address(eth_abi.decode_single('address', bytes.fromhex(strip_0x(r))))
|
||||
except ValueError as e:
|
||||
logg.critical('lookup failed for token {}: {}'.format(token_symbol, e))
|
||||
sys.exit(1)
|
||||
logg.info('found token address {}'.format(sarafu_token_address))
|
||||
|
||||
syncer_backend = MemBackend(chain_str, 0)
|
||||
|
||||
if block_offset == -1:
|
||||
o = block_latest()
|
||||
r = conn.do(o)
|
||||
block_offset = int(strip_0x(r), 16) + 1
|
||||
#
|
||||
# addresses = {}
|
||||
# f = open('{}/addresses.csv'.format(user_dir, 'r'))
|
||||
# while True:
|
||||
# l = f.readline()
|
||||
# if l == None:
|
||||
# break
|
||||
# r = l.split(',')
|
||||
# try:
|
||||
# k = r[0]
|
||||
# v = r[1].rstrip()
|
||||
# addresses[k] = v
|
||||
# sys.stdout.write('loading address mapping {} -> {}'.format(k, v).ljust(200) + "\r")
|
||||
# except IndexError as e:
|
||||
# break
|
||||
# f.close()
|
||||
|
||||
# TODO get decimals from token
|
||||
balances = {}
|
||||
f = open('{}/balances.csv'.format(user_dir, 'r'))
|
||||
remove_zeros = 10**6
|
||||
i = 0
|
||||
while True:
|
||||
l = f.readline()
|
||||
if l == None:
|
||||
break
|
||||
r = l.split(',')
|
||||
try:
|
||||
address = to_checksum_address(r[0])
|
||||
sys.stdout.write('loading balance {} {} {}'.format(i, address, r[1]).ljust(200) + "\r")
|
||||
except ValueError:
|
||||
break
|
||||
balance = int(int(r[1].rstrip()) / remove_zeros)
|
||||
balances[address] = balance
|
||||
i += 1
|
||||
|
||||
f.close()
|
||||
|
||||
syncer_backend.set(block_offset, 0)
|
||||
syncer = HeadSyncer(syncer_backend, block_callback=progress_callback)
|
||||
handler = Handler(conn, chain_spec, user_dir, balances, sarafu_token_address, signer, gas_oracle, nonce_oracle)
|
||||
syncer.add_filter(handler)
|
||||
syncer.loop(1, conn)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -1,248 +0,0 @@
|
||||
# standard imports
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import logging
|
||||
import argparse
|
||||
import uuid
|
||||
import datetime
|
||||
import time
|
||||
import phonenumbers
|
||||
from glob import glob
|
||||
|
||||
# external imports
|
||||
import confini
|
||||
from hexathon import (
|
||||
add_0x,
|
||||
strip_0x,
|
||||
)
|
||||
from cic_types.models.person import Person
|
||||
from chainlib.eth.address import to_checksum_address
|
||||
from chainlib.chain import ChainSpec
|
||||
from chainlib.eth.connection import EthHTTPConnection
|
||||
from chainlib.eth.gas import RPCGasOracle
|
||||
from chainlib.eth.nonce import RPCNonceOracle
|
||||
from cic_types.processor import generate_metadata_pointer
|
||||
from eth_accounts_index.registry import AccountRegistry
|
||||
from eth_contract_registry import Registry
|
||||
from crypto_dev_signer.keystore.dict import DictKeystore
|
||||
from crypto_dev_signer.eth.signer.defaultsigner import ReferenceSigner as EIP155Signer
|
||||
from crypto_dev_signer.keystore.keyfile import to_dict as to_keyfile_dict
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
default_config_dir = '/usr/local/etc/cic'
|
||||
|
||||
argparser = argparse.ArgumentParser()
|
||||
argparser.add_argument('-p', '--provider', dest='p', default='http://localhost:8545', type=str, help='Web3 provider url (http only)')
|
||||
argparser.add_argument('-y', '--key-file', dest='y', type=str, help='Ethereum keystore file to use for signing')
|
||||
argparser.add_argument('-c', type=str, default=default_config_dir, help='config file')
|
||||
argparser.add_argument('--old-chain-spec', type=str, dest='old_chain_spec', default='evm:oldchain:1', help='chain spec')
|
||||
argparser.add_argument('-i', '--chain-spec', dest='i', type=str, help='Chain specification string')
|
||||
argparser.add_argument('-r', '--registry', dest='r', type=str, help='Contract registry address')
|
||||
argparser.add_argument('--batch-size', dest='batch_size', default=50, type=int, help='burst size of sending transactions to node')
|
||||
argparser.add_argument('--batch-delay', dest='batch_delay', default=2, type=int, help='seconds delay between batches')
|
||||
argparser.add_argument('-v', action='store_true', help='Be verbose')
|
||||
argparser.add_argument('-vv', action='store_true', help='Be more verbose')
|
||||
argparser.add_argument('user_dir', type=str, help='path to users export dir tree')
|
||||
args = argparser.parse_args()
|
||||
|
||||
if args.v:
|
||||
logg.setLevel(logging.INFO)
|
||||
elif args.vv:
|
||||
logg.setLevel(logging.DEBUG)
|
||||
|
||||
config_dir = args.c
|
||||
config = confini.Config(config_dir, os.environ.get('CONFINI_ENV_PREFIX'))
|
||||
config.process()
|
||||
args_override = {
|
||||
'CIC_REGISTRY_ADDRESS': getattr(args, 'r'),
|
||||
'CIC_CHAIN_SPEC': getattr(args, 'i'),
|
||||
}
|
||||
config.dict_override(args_override, 'cli')
|
||||
config.add(args.user_dir, '_USERDIR', True)
|
||||
|
||||
user_new_dir = os.path.join(args.user_dir, 'new')
|
||||
os.makedirs(user_new_dir)
|
||||
|
||||
meta_dir = os.path.join(args.user_dir, 'meta')
|
||||
os.makedirs(meta_dir)
|
||||
|
||||
custom_dir = os.path.join(args.user_dir, 'custom')
|
||||
os.makedirs(custom_dir)
|
||||
os.makedirs(os.path.join(custom_dir, 'new'))
|
||||
os.makedirs(os.path.join(custom_dir, 'meta'))
|
||||
|
||||
phone_dir = os.path.join(args.user_dir, 'phone')
|
||||
os.makedirs(os.path.join(phone_dir, 'meta'))
|
||||
|
||||
user_old_dir = os.path.join(args.user_dir, 'old')
|
||||
os.stat(user_old_dir)
|
||||
|
||||
txs_dir = os.path.join(args.user_dir, 'txs')
|
||||
os.makedirs(txs_dir)
|
||||
|
||||
user_dir = args.user_dir
|
||||
|
||||
chain_spec = ChainSpec.from_chain_str(config.get('CIC_CHAIN_SPEC'))
|
||||
chain_str = str(chain_spec)
|
||||
|
||||
old_chain_spec = ChainSpec.from_chain_str(args.old_chain_spec)
|
||||
old_chain_str = str(old_chain_spec)
|
||||
|
||||
batch_size = args.batch_size
|
||||
batch_delay = args.batch_delay
|
||||
|
||||
rpc = EthHTTPConnection(args.p)
|
||||
|
||||
signer_address = None
|
||||
keystore = DictKeystore()
|
||||
if args.y != None:
|
||||
logg.debug('loading keystore file {}'.format(args.y))
|
||||
signer_address = keystore.import_keystore_file(args.y)
|
||||
logg.debug('now have key for signer address {}'.format(signer_address))
|
||||
signer = EIP155Signer(keystore)
|
||||
|
||||
nonce_oracle = RPCNonceOracle(signer_address, rpc)
|
||||
|
||||
registry = Registry(chain_spec)
|
||||
o = registry.address_of(config.get('CIC_REGISTRY_ADDRESS'), 'AccountRegistry')
|
||||
r = rpc.do(o)
|
||||
account_registry_address = registry.parse_address_of(r)
|
||||
logg.info('using account registry {}'.format(account_registry_address))
|
||||
|
||||
keyfile_dir = os.path.join(config.get('_USERDIR'), 'keystore')
|
||||
os.makedirs(keyfile_dir)
|
||||
|
||||
def register_eth(i, u):
|
||||
|
||||
address_hex = keystore.new()
|
||||
address = add_0x(to_checksum_address(address_hex))
|
||||
|
||||
gas_oracle = RPCGasOracle(rpc, code_callback=AccountRegistry.gas)
|
||||
c = AccountRegistry(chain_spec, signer=signer, nonce_oracle=nonce_oracle, gas_oracle=gas_oracle)
|
||||
(tx_hash_hex, o) = c.add(account_registry_address, signer_address, address)
|
||||
logg.debug('o {}'.format(o))
|
||||
rpc.do(o)
|
||||
|
||||
pk = keystore.get(address)
|
||||
keyfile_content = to_keyfile_dict(pk, 'foo')
|
||||
keyfile_path = os.path.join(keyfile_dir, '{}.json'.format(address))
|
||||
f = open(keyfile_path, 'w')
|
||||
json.dump(keyfile_content, f)
|
||||
f.close()
|
||||
|
||||
logg.debug('[{}] register eth {} {} tx {} keyfile {}'.format(i, u, address, tx_hash_hex, keyfile_path))
|
||||
|
||||
return address
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
user_tags = {}
|
||||
f = open(os.path.join(user_dir, 'tags.csv'), 'r')
|
||||
while True:
|
||||
r = f.readline().rstrip()
|
||||
if len(r) == 0:
|
||||
break
|
||||
(old_address, tags_csv) = r.split(':')
|
||||
old_address = strip_0x(old_address)
|
||||
user_tags[old_address] = tags_csv.split(',')
|
||||
logg.debug('read tags {} for old address {}'.format(user_tags[old_address], old_address))
|
||||
|
||||
i = 0
|
||||
j = 0
|
||||
for x in os.walk(user_old_dir):
|
||||
for y in x[2]:
|
||||
if y[len(y)-5:] != '.json':
|
||||
continue
|
||||
filepath = os.path.join(x[0], y)
|
||||
f = open(filepath, 'r')
|
||||
try:
|
||||
o = json.load(f)
|
||||
except json.decoder.JSONDecodeError as e:
|
||||
f.close()
|
||||
logg.error('load error for {}: {}'.format(y, e))
|
||||
continue
|
||||
f.close()
|
||||
u = Person.deserialize(o)
|
||||
logg.debug('u {}'.format(o))
|
||||
|
||||
new_address = register_eth(i, u)
|
||||
if u.identities.get('evm') == None:
|
||||
u.identities['evm'] = {}
|
||||
sub_chain_str = '{}:{}'.format(chain_spec.common_name(), chain_spec.network_id())
|
||||
u.identities['evm'][sub_chain_str] = [new_address]
|
||||
|
||||
new_address_clean = strip_0x(new_address)
|
||||
filepath = os.path.join(
|
||||
user_new_dir,
|
||||
new_address_clean[:2].upper(),
|
||||
new_address_clean[2:4].upper(),
|
||||
new_address_clean.upper() + '.json',
|
||||
)
|
||||
os.makedirs(os.path.dirname(filepath), exist_ok=True)
|
||||
|
||||
o = u.serialize()
|
||||
f = open(filepath, 'w')
|
||||
f.write(json.dumps(o))
|
||||
f.close()
|
||||
|
||||
meta_key = generate_metadata_pointer(bytes.fromhex(new_address_clean), ':cic.person')
|
||||
meta_filepath = os.path.join(meta_dir, '{}.json'.format(new_address_clean.upper()))
|
||||
os.symlink(os.path.realpath(filepath), meta_filepath)
|
||||
|
||||
phone_object = phonenumbers.parse(u.tel)
|
||||
phone = phonenumbers.format_number(phone_object, phonenumbers.PhoneNumberFormat.E164)
|
||||
logg.debug('>>>>> Using phone {}'.format(phone))
|
||||
meta_phone_key = generate_metadata_pointer(phone.encode('utf-8'), ':cic.phone')
|
||||
meta_phone_filepath = os.path.join(phone_dir, 'meta', meta_phone_key)
|
||||
|
||||
filepath = os.path.join(
|
||||
phone_dir,
|
||||
'new',
|
||||
meta_phone_key[:2].upper(),
|
||||
meta_phone_key[2:4].upper(),
|
||||
meta_phone_key.upper(),
|
||||
)
|
||||
os.makedirs(os.path.dirname(filepath), exist_ok=True)
|
||||
|
||||
f = open(filepath, 'w')
|
||||
f.write(to_checksum_address(new_address_clean))
|
||||
f.close()
|
||||
|
||||
os.symlink(os.path.realpath(filepath), meta_phone_filepath)
|
||||
|
||||
|
||||
# custom data
|
||||
custom_key = generate_metadata_pointer(phone.encode('utf-8'), ':cic.custom')
|
||||
custom_filepath = os.path.join(custom_dir, 'meta', custom_key)
|
||||
|
||||
filepath = os.path.join(
|
||||
custom_dir,
|
||||
'new',
|
||||
custom_key[:2].upper(),
|
||||
custom_key[2:4].upper(),
|
||||
custom_key.upper() + '.json',
|
||||
)
|
||||
os.makedirs(os.path.dirname(filepath), exist_ok=True)
|
||||
|
||||
sub_old_chain_str = '{}:{}'.format(old_chain_spec.common_name(), old_chain_spec.network_id())
|
||||
f = open(filepath, 'w')
|
||||
k = u.identities['evm'][sub_old_chain_str][0]
|
||||
tag_data = {'tags': user_tags[strip_0x(k)]}
|
||||
f.write(json.dumps(tag_data))
|
||||
f.close()
|
||||
|
||||
os.symlink(os.path.realpath(filepath), custom_filepath)
|
||||
|
||||
i += 1
|
||||
sys.stdout.write('imported {} {}'.format(i, u).ljust(200) + "\r")
|
||||
|
||||
j += 1
|
||||
if j == batch_size:
|
||||
time.sleep(batch_delay)
|
||||
j = 0
|
||||
|
||||
#fi.close()
|
||||
3851
apps/contract-migration/scripts/package-lock.json
generated
3851
apps/contract-migration/scripts/package-lock.json
generated
File diff suppressed because it is too large
Load Diff
@@ -1,7 +0,0 @@
|
||||
{
|
||||
"dependencies": {
|
||||
"cic-client-meta": "^0.0.7-alpha.8",
|
||||
"crdt-meta": "0.0.8",
|
||||
"vcard-parser": "^1.0.0"
|
||||
}
|
||||
}
|
||||
@@ -1,5 +0,0 @@
|
||||
cic-base[full_graph]==0.1.2b9
|
||||
sarafu-faucet==0.0.3a3
|
||||
cic-eth==0.11.0b13
|
||||
cic-types==0.1.0a11
|
||||
crypto-dev-signer==0.4.14b3
|
||||
@@ -1,536 +0,0 @@
|
||||
# standard imports
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import urllib
|
||||
import urllib.request
|
||||
import uuid
|
||||
|
||||
# external imports
|
||||
import celery
|
||||
import confini
|
||||
import eth_abi
|
||||
from chainlib.chain import ChainSpec
|
||||
from chainlib.eth.address import to_checksum_address
|
||||
from chainlib.eth.connection import EthHTTPConnection
|
||||
from chainlib.eth.constant import ZERO_ADDRESS
|
||||
from chainlib.eth.gas import (
|
||||
OverrideGasOracle,
|
||||
balance,
|
||||
)
|
||||
from chainlib.eth.tx import TxFactory
|
||||
from chainlib.hash import keccak256_string_to_hex
|
||||
from chainlib.jsonrpc import jsonrpc_template
|
||||
from cic_types.models.person import (
|
||||
Person,
|
||||
generate_metadata_pointer,
|
||||
)
|
||||
from erc20_faucet import Faucet
|
||||
from eth_erc20 import ERC20
|
||||
from hexathon.parse import strip_0x, add_0x
|
||||
|
||||
logging.basicConfig(level=logging.WARNING)
|
||||
logg = logging.getLogger()
|
||||
|
||||
config_dir = '/usr/local/etc/cic-syncer'
|
||||
|
||||
custodial_tests = [
|
||||
'local_key',
|
||||
'gas',
|
||||
'faucet',
|
||||
'ussd'
|
||||
]
|
||||
|
||||
metadata_tests = [
|
||||
'metadata',
|
||||
'metadata_phone',
|
||||
]
|
||||
|
||||
eth_tests = [
|
||||
'accounts_index',
|
||||
'balance',
|
||||
]
|
||||
|
||||
phone_tests = [
|
||||
'ussd',
|
||||
'ussd_pins'
|
||||
]
|
||||
|
||||
all_tests = eth_tests + custodial_tests + metadata_tests + phone_tests
|
||||
|
||||
argparser = argparse.ArgumentParser(description='daemon that monitors transactions in new blocks')
|
||||
argparser.add_argument('-p', '--provider', dest='p', type=str, help='chain rpc provider address')
|
||||
argparser.add_argument('-c', type=str, default=config_dir, help='config root to use')
|
||||
argparser.add_argument('--old-chain-spec', type=str, dest='old_chain_spec', default='evm:oldchain:1', help='chain spec')
|
||||
argparser.add_argument('-i', '--chain-spec', type=str, dest='i', help='chain spec')
|
||||
argparser.add_argument('--meta-provider', type=str, dest='meta_provider', default='http://localhost:63380', help='cic-meta url')
|
||||
argparser.add_argument('--ussd-provider', type=str, dest='ussd_provider', default='http://localhost:63315', help='cic-ussd url')
|
||||
argparser.add_argument('--skip-custodial', dest='skip_custodial', action='store_true', help='skip all custodial verifications')
|
||||
argparser.add_argument('--exclude', action='append', type=str, default=[], help='skip specified verification')
|
||||
argparser.add_argument('--include', action='append', type=str, help='include specified verification')
|
||||
argparser.add_argument('-r', '--registry-address', type=str, dest='r', help='CIC Registry address')
|
||||
argparser.add_argument('--env-prefix', default=os.environ.get('CONFINI_ENV_PREFIX'), dest='env_prefix', type=str, help='environment prefix for variables to overwrite configuration')
|
||||
argparser.add_argument('-x', '--exit-on-error', dest='x', action='store_true', help='Halt exection on error')
|
||||
argparser.add_argument('-v', help='be verbose', action='store_true')
|
||||
argparser.add_argument('-vv', help='be more verbose', action='store_true')
|
||||
argparser.add_argument('user_dir', type=str, help='user export directory')
|
||||
args = argparser.parse_args(sys.argv[1:])
|
||||
|
||||
if args.v == True:
|
||||
logging.getLogger().setLevel(logging.INFO)
|
||||
elif args.vv == True:
|
||||
logging.getLogger().setLevel(logging.DEBUG)
|
||||
|
||||
config_dir = os.path.join(args.c)
|
||||
os.makedirs(config_dir, 0o777, True)
|
||||
config = confini.Config(config_dir, args.env_prefix)
|
||||
config.process()
|
||||
# override args
|
||||
args_override = {
|
||||
'CIC_CHAIN_SPEC': getattr(args, 'i'),
|
||||
'ETH_PROVIDER': getattr(args, 'p'),
|
||||
'CIC_REGISTRY_ADDRESS': getattr(args, 'r'),
|
||||
}
|
||||
config.dict_override(args_override, 'cli flag')
|
||||
config.censor('PASSWORD', 'DATABASE')
|
||||
config.censor('PASSWORD', 'SSL')
|
||||
config.add(args.meta_provider, '_META_PROVIDER', True)
|
||||
config.add(args.ussd_provider, '_USSD_PROVIDER', True)
|
||||
|
||||
logg.debug('config loaded from {}:\n{}'.format(config_dir, config))
|
||||
|
||||
celery_app = celery.Celery(backend=config.get('CELERY_RESULT_URL'), broker=config.get('CELERY_BROKER_URL'))
|
||||
|
||||
chain_spec = ChainSpec.from_chain_str(config.get('CIC_CHAIN_SPEC'))
|
||||
chain_str = str(chain_spec)
|
||||
old_chain_spec = ChainSpec.from_chain_str(args.old_chain_spec)
|
||||
old_chain_str = str(old_chain_spec)
|
||||
user_dir = args.user_dir # user_out_dir from import_users.py
|
||||
exit_on_error = args.x
|
||||
|
||||
active_tests = []
|
||||
exclude = []
|
||||
include = args.include
|
||||
if args.include == None:
|
||||
include = all_tests
|
||||
for t in args.exclude:
|
||||
if t not in all_tests:
|
||||
raise ValueError('Cannot exclude unknown verification "{}"'.format(t))
|
||||
exclude.append(t)
|
||||
if args.skip_custodial:
|
||||
logg.info('will skip all custodial verifications ({})'.format(','.join(custodial_tests)))
|
||||
for t in custodial_tests:
|
||||
if t not in exclude:
|
||||
exclude.append(t)
|
||||
for t in include:
|
||||
if t not in all_tests:
|
||||
raise ValueError('Cannot include unknown verification "{}"'.format(t))
|
||||
if t not in exclude:
|
||||
active_tests.append(t)
|
||||
logg.info('will perform verification "{}"'.format(t))
|
||||
|
||||
api = None
|
||||
for t in custodial_tests:
|
||||
if t in active_tests:
|
||||
from cic_eth.api.api_admin import AdminApi
|
||||
api = AdminApi(None)
|
||||
logg.info('activating custodial module'.format(t))
|
||||
break
|
||||
|
||||
cols = os.get_terminal_size().columns
|
||||
|
||||
|
||||
def to_terminalwidth(s):
|
||||
ss = s.ljust(int(cols)-1)
|
||||
ss += "\r"
|
||||
return ss
|
||||
|
||||
def default_outfunc(s):
|
||||
ss = to_terminalwidth(s)
|
||||
sys.stdout.write(ss)
|
||||
outfunc = default_outfunc
|
||||
if logg.isEnabledFor(logging.DEBUG):
|
||||
outfunc = logg.debug
|
||||
|
||||
|
||||
def send_ussd_request(address, data_dir):
|
||||
upper_address = strip_0x(address).upper()
|
||||
f = open(os.path.join(
|
||||
data_dir,
|
||||
'new',
|
||||
upper_address[:2],
|
||||
upper_address[2:4],
|
||||
upper_address + '.json',
|
||||
), 'r'
|
||||
)
|
||||
o = json.load(f)
|
||||
f.close()
|
||||
|
||||
p = Person.deserialize(o)
|
||||
phone = p.tel
|
||||
|
||||
session = uuid.uuid4().hex
|
||||
data = {
|
||||
'sessionId': session,
|
||||
'serviceCode': config.get('APP_SERVICE_CODE'),
|
||||
'phoneNumber': phone,
|
||||
'text': '',
|
||||
}
|
||||
|
||||
req = urllib.request.Request(config.get('_USSD_PROVIDER'))
|
||||
data_str = json.dumps(data)
|
||||
data_bytes = data_str.encode('utf-8')
|
||||
req.add_header('Content-Type', 'application/json')
|
||||
req.data = data_bytes
|
||||
response = urllib.request.urlopen(req)
|
||||
return response.read().decode('utf-8')
|
||||
|
||||
|
||||
class VerifierState:
|
||||
|
||||
def __init__(self, item_keys, active_tests=None):
|
||||
self.items = {}
|
||||
for k in item_keys:
|
||||
self.items[k] = 0
|
||||
if active_tests == None:
|
||||
self.active_tests = copy.copy(item_keys)
|
||||
else:
|
||||
self.active_tests = copy.copy(active_tests)
|
||||
|
||||
|
||||
def poke(self, item_key):
|
||||
self.items[item_key] += 1
|
||||
|
||||
|
||||
def __str__(self):
|
||||
r = ''
|
||||
for k in self.items.keys():
|
||||
if k in self.active_tests:
|
||||
r += '{}: {}\n'.format(k, self.items[k])
|
||||
else:
|
||||
r += '{}: skipped\n'.format(k)
|
||||
return r
|
||||
|
||||
|
||||
class VerifierError(Exception):
|
||||
|
||||
def __init__(self, e, c):
|
||||
super(VerifierError, self).__init__(e)
|
||||
self.c = c
|
||||
|
||||
|
||||
def __str__(self):
|
||||
super_error = super(VerifierError, self).__str__()
|
||||
return '[{}] {}'.format(self.c, super_error)
|
||||
|
||||
|
||||
class Verifier:
|
||||
|
||||
# TODO: what an awful function signature
|
||||
def __init__(self, conn, cic_eth_api, gas_oracle, chain_spec, index_address, token_address, faucet_address, data_dir, exit_on_error=False):
|
||||
self.conn = conn
|
||||
self.gas_oracle = gas_oracle
|
||||
self.chain_spec = chain_spec
|
||||
self.index_address = index_address
|
||||
self.token_address = token_address
|
||||
self.faucet_address = faucet_address
|
||||
self.erc20_tx_factory = ERC20(chain_spec, gas_oracle=gas_oracle)
|
||||
self.tx_factory = TxFactory(chain_spec, gas_oracle=gas_oracle)
|
||||
self.api = cic_eth_api
|
||||
self.data_dir = data_dir
|
||||
self.exit_on_error = exit_on_error
|
||||
self.faucet_tx_factory = Faucet(chain_spec, gas_oracle=gas_oracle)
|
||||
|
||||
verifymethods = []
|
||||
for k in dir(self):
|
||||
if len(k) > 7 and k[:7] == 'verify_':
|
||||
logg.debug('verifier has verify method {}'.format(k))
|
||||
verifymethods.append(k[7:])
|
||||
|
||||
self.state = VerifierState(verifymethods, active_tests=active_tests)
|
||||
|
||||
|
||||
def verify_accounts_index(self, address, balance=None):
|
||||
tx = self.tx_factory.template(ZERO_ADDRESS, self.index_address)
|
||||
data = keccak256_string_to_hex('have(address)')[:8]
|
||||
data += eth_abi.encode_single('address', address).hex()
|
||||
tx = self.tx_factory.set_code(tx, data)
|
||||
tx = self.tx_factory.normalize(tx)
|
||||
o = jsonrpc_template()
|
||||
o['method'] = 'eth_call'
|
||||
o['params'].append(tx)
|
||||
r = self.conn.do(o)
|
||||
logg.debug('index check for {}: {}'.format(address, r))
|
||||
n = eth_abi.decode_single('uint256', bytes.fromhex(strip_0x(r)))
|
||||
if n != 1:
|
||||
raise VerifierError(n, 'accounts index')
|
||||
|
||||
|
||||
def verify_balance(self, address, balance):
|
||||
o = self.erc20_tx_factory.balance(self.token_address, address)
|
||||
r = self.conn.do(o)
|
||||
actual_balance = int(strip_0x(r), 16)
|
||||
balance = int(balance / 1000000) * 1000000
|
||||
logg.debug('balance for {}: {}'.format(address, balance))
|
||||
if balance != actual_balance:
|
||||
raise VerifierError((actual_balance, balance), 'balance')
|
||||
|
||||
|
||||
def verify_local_key(self, address, balance=None):
|
||||
t = self.api.have_account(address, self.chain_spec)
|
||||
r = t.get()
|
||||
logg.debug('verify local key result {}'.format(r))
|
||||
if r != address:
|
||||
raise VerifierError((address, r), 'local key')
|
||||
|
||||
|
||||
def verify_gas(self, address, balance_token=None):
|
||||
o = balance(address)
|
||||
r = self.conn.do(o)
|
||||
logg.debug('wtf {}'.format(r))
|
||||
actual_balance = int(strip_0x(r), 16)
|
||||
if actual_balance == 0:
|
||||
raise VerifierError((address, actual_balance), 'gas')
|
||||
|
||||
|
||||
def verify_faucet(self, address, balance_token=None):
|
||||
o = self.faucet_tx_factory.usable_for(self.faucet_address, address)
|
||||
r = self.conn.do(o)
|
||||
if self.faucet_tx_factory.parse_usable_for(r):
|
||||
raise VerifierError((address, r), 'faucet')
|
||||
|
||||
|
||||
def verify_metadata(self, address, balance=None):
|
||||
k = generate_metadata_pointer(bytes.fromhex(strip_0x(address)), ':cic.person')
|
||||
url = os.path.join(config.get('_META_PROVIDER'), k)
|
||||
logg.debug('verify metadata url {}'.format(url))
|
||||
try:
|
||||
res = urllib.request.urlopen(url)
|
||||
except urllib.error.HTTPError as e:
|
||||
raise VerifierError(
|
||||
'({}) {}'.format(url, e),
|
||||
'metadata (person)',
|
||||
)
|
||||
b = res.read()
|
||||
o_retrieved = json.loads(b.decode('utf-8'))
|
||||
|
||||
upper_address = strip_0x(address).upper()
|
||||
f = open(os.path.join(
|
||||
self.data_dir,
|
||||
'new',
|
||||
upper_address[:2],
|
||||
upper_address[2:4],
|
||||
upper_address + '.json',
|
||||
), 'r'
|
||||
)
|
||||
o_original = json.load(f)
|
||||
f.close()
|
||||
|
||||
if o_original != o_retrieved:
|
||||
raise VerifierError(o_retrieved, 'metadata (person)')
|
||||
|
||||
|
||||
def verify_metadata_phone(self, address, balance=None):
|
||||
upper_address = strip_0x(address).upper()
|
||||
f = open(os.path.join(
|
||||
self.data_dir,
|
||||
'new',
|
||||
upper_address[:2],
|
||||
upper_address[2:4],
|
||||
upper_address + '.json',
|
||||
), 'r'
|
||||
)
|
||||
o = json.load(f)
|
||||
f.close()
|
||||
|
||||
p = Person.deserialize(o)
|
||||
|
||||
k = generate_metadata_pointer(p.tel.encode('utf-8'), ':cic.phone')
|
||||
url = os.path.join(config.get('_META_PROVIDER'), k)
|
||||
logg.debug('verify metadata phone url {}'.format(url))
|
||||
try:
|
||||
res = urllib.request.urlopen(url)
|
||||
except urllib.error.HTTPError as e:
|
||||
raise VerifierError(
|
||||
'({}) {}'.format(url, e),
|
||||
'metadata (phone)',
|
||||
)
|
||||
b = res.read()
|
||||
address_recovered = json.loads(b.decode('utf-8'))
|
||||
address_recovered = address_recovered.replace('"', '')
|
||||
|
||||
try:
|
||||
upper_address_recovered = strip_0x(address_recovered).upper()
|
||||
except ValueError:
|
||||
raise VerifierError(address_recovered, 'metadata (phone) address {} address recovered {}'.format(address, address_recovered))
|
||||
|
||||
if upper_address != upper_address_recovered:
|
||||
raise VerifierError(address_recovered, 'metadata (phone)')
|
||||
|
||||
|
||||
def verify_ussd(self, address, balance=None):
|
||||
response_data = send_ussd_request(address, self.data_dir)
|
||||
state = response_data[:3]
|
||||
out = response_data[4:]
|
||||
m = '{} {}'.format(state, out[:7])
|
||||
if m != 'CON Welcome':
|
||||
raise VerifierError(response_data, 'ussd')
|
||||
|
||||
def verify_ussd_pins(self, address, balance):
|
||||
response_data = send_ussd_request(address, self.data_dir)
|
||||
if response_data[:11] != 'CON Balance':
|
||||
raise VerifierError(response_data, 'pins')
|
||||
|
||||
|
||||
def verify(self, address, balance, debug_stem=None):
|
||||
|
||||
for k in active_tests:
|
||||
s = '{} {}'.format(debug_stem, k)
|
||||
outfunc(s)
|
||||
try:
|
||||
m = getattr(self, 'verify_{}'.format(k))
|
||||
m(address, balance)
|
||||
except VerifierError as e:
|
||||
logline = 'verification {} failed for {}: {}'.format(k, address, str(e))
|
||||
if self.exit_on_error:
|
||||
logg.critical(logline)
|
||||
sys.exit(1)
|
||||
logg.error(logline)
|
||||
self.state.poke(k)
|
||||
|
||||
|
||||
def __str__(self):
|
||||
return str(self.state)
|
||||
|
||||
|
||||
def main():
|
||||
global chain_str, block_offset, user_dir
|
||||
|
||||
conn = EthHTTPConnection(config.get('ETH_PROVIDER'))
|
||||
gas_oracle = OverrideGasOracle(conn=conn, limit=8000000)
|
||||
|
||||
# Get Token registry address
|
||||
txf = TxFactory(chain_spec, signer=None, gas_oracle=gas_oracle, nonce_oracle=None)
|
||||
tx = txf.template(ZERO_ADDRESS, config.get('CIC_REGISTRY_ADDRESS'))
|
||||
|
||||
# TODO: replace with cic-eth-registry
|
||||
registry_addressof_method = keccak256_string_to_hex('addressOf(bytes32)')[:8]
|
||||
data = add_0x(registry_addressof_method)
|
||||
data += eth_abi.encode_single('bytes32', b'TokenRegistry').hex()
|
||||
txf.set_code(tx, data)
|
||||
|
||||
o = jsonrpc_template()
|
||||
o['method'] = 'eth_call'
|
||||
o['params'].append(txf.normalize(tx))
|
||||
o['params'].append('latest')
|
||||
r = conn.do(o)
|
||||
token_index_address = to_checksum_address(eth_abi.decode_single('address', bytes.fromhex(strip_0x(r))))
|
||||
logg.info('found token index address {}'.format(token_index_address))
|
||||
|
||||
data = add_0x(registry_addressof_method)
|
||||
data += eth_abi.encode_single('bytes32', b'AccountRegistry').hex()
|
||||
txf.set_code(tx, data)
|
||||
|
||||
o = jsonrpc_template()
|
||||
o['method'] = 'eth_call'
|
||||
o['params'].append(txf.normalize(tx))
|
||||
o['params'].append('latest')
|
||||
r = conn.do(o)
|
||||
account_index_address = to_checksum_address(eth_abi.decode_single('address', bytes.fromhex(strip_0x(r))))
|
||||
logg.info('found account index address {}'.format(account_index_address))
|
||||
|
||||
data = add_0x(registry_addressof_method)
|
||||
data += eth_abi.encode_single('bytes32', b'Faucet').hex()
|
||||
txf.set_code(tx, data)
|
||||
|
||||
o = jsonrpc_template()
|
||||
o['method'] = 'eth_call'
|
||||
o['params'].append(txf.normalize(tx))
|
||||
o['params'].append('latest')
|
||||
r = conn.do(o)
|
||||
faucet_address = to_checksum_address(eth_abi.decode_single('address', bytes.fromhex(strip_0x(r))))
|
||||
logg.info('found faucet {}'.format(faucet_address))
|
||||
|
||||
|
||||
|
||||
# Get Sarafu token address
|
||||
tx = txf.template(ZERO_ADDRESS, token_index_address)
|
||||
data = add_0x(registry_addressof_method)
|
||||
h = hashlib.new('sha256')
|
||||
h.update(b'SRF')
|
||||
z = h.digest()
|
||||
data += eth_abi.encode_single('bytes32', z).hex()
|
||||
txf.set_code(tx, data)
|
||||
o = jsonrpc_template()
|
||||
o['method'] = 'eth_call'
|
||||
o['params'].append(txf.normalize(tx))
|
||||
o['params'].append('latest')
|
||||
r = conn.do(o)
|
||||
sarafu_token_address = to_checksum_address(eth_abi.decode_single('address', bytes.fromhex(strip_0x(r))))
|
||||
logg.info('found token address {}'.format(sarafu_token_address))
|
||||
|
||||
balances = {}
|
||||
f = open('{}/balances.csv'.format(user_dir, 'r'))
|
||||
i = 0
|
||||
while True:
|
||||
l = f.readline()
|
||||
if l == None:
|
||||
break
|
||||
r = l.split(',')
|
||||
try:
|
||||
address = to_checksum_address(r[0])
|
||||
#sys.stdout.write('loading balance {} {}'.format(i, address).ljust(200) + "\r")
|
||||
outfunc('loading balance {} {}'.format(i, address)) #.ljust(200))
|
||||
except ValueError:
|
||||
break
|
||||
balance = int(r[1].rstrip())
|
||||
balances[address] = balance
|
||||
i += 1
|
||||
|
||||
f.close()
|
||||
|
||||
verifier = Verifier(conn, api, gas_oracle, chain_spec, account_index_address, sarafu_token_address, faucet_address, user_dir, exit_on_error)
|
||||
|
||||
user_new_dir = os.path.join(user_dir, 'new')
|
||||
i = 0
|
||||
for x in os.walk(user_new_dir):
|
||||
for y in x[2]:
|
||||
if y[len(y)-5:] != '.json':
|
||||
continue
|
||||
filepath = os.path.join(x[0], y)
|
||||
f = open(filepath, 'r')
|
||||
try:
|
||||
o = json.load(f)
|
||||
except json.decoder.JSONDecodeError as e:
|
||||
f.close()
|
||||
logg.error('load error for {}: {}'.format(y, e))
|
||||
continue
|
||||
f.close()
|
||||
|
||||
u = Person.deserialize(o)
|
||||
#logg.debug('data {}'.format(u.identities['evm']))
|
||||
|
||||
subchain_str = '{}:{}'.format(chain_spec.common_name(), chain_spec.network_id())
|
||||
new_address = u.identities['evm'][subchain_str][0]
|
||||
subchain_str = '{}:{}'.format(old_chain_spec.common_name(), old_chain_spec.network_id())
|
||||
old_address = u.identities['evm'][subchain_str][0]
|
||||
balance = 0
|
||||
try:
|
||||
balance = balances[old_address]
|
||||
except KeyError:
|
||||
logg.info('no old balance found for {}, assuming 0'.format(old_address))
|
||||
|
||||
s = 'checking {}: {} -> {} = {}'.format(i, old_address, new_address, balance)
|
||||
|
||||
verifier.verify(new_address, balance, debug_stem=s)
|
||||
i += 1
|
||||
|
||||
print(verifier)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user