Commit Graph

40 Commits

Author SHA1 Message Date
Tomasz Drwięga
9438afde32 Recently rejected cache for transaction queue (#9005)
* Store recently rejected transactions.

* Don't cache AlreadyImported rejections.

* Make the size of transaction verification queue dependent on pool size.

* Add a test for recently rejected.

* Fix logging for recently rejected.

* Make rejection cache smaller.

* obsolete test removed

* obsolete test removed

* Construct cache with_capacity.
2018-07-07 16:17:50 +01:00
André Silva
7bf16cce1b [beta] Backports (#8785)
* Fix light sync with initial validator-set contract (#8528)

* Fix #8468

* Use U256::max_value() instead

* Fix again

* Also change initial transaction gas

* Resumable warp-sync / Seed downloaded snapshots (#8544)

* Start dividing sync chain : first supplier method

* WIP - updated chain sync supplier

* Finish refactoring the Chain Sync Supplier

* Create Chain Sync Requester

* Add Propagator for Chain Sync

* Add the Chain Sync Handler

* Move tests from mod -> handler

* Move tests to propagator

* Refactor SyncRequester arguments

* Refactoring peer fork header handler

* Fix wrong highest block number in snapshot sync

* Small refactor...

* Resume warp-sync downloaded chunks

* Add comments

* Refactoring the previous chunks import

* Fix tests

* Address PR grumbles

* Fix not seeding current snapshot

* Address PR Grumbles

* Address PR grumble

* Retry failed CI job

* Update SnapshotService readiness check
Fix restoration locking issue for previous chunks restoration

* Fix tests

* Fix tests

* Fix test

* Early abort importing previous chunks

* PR Grumbles

* Update Gitlab CI config

* SyncState back to Waiting when Manifest peers disconnect

* Move fix

* Better fix

* Revert GitLab CI changes

* Fix Warning

* Refactor resuming snapshots

* Fix string construction

* Revert "Refactor resuming snapshots"

This reverts commit 75fd4b553a38e4a49dc5d6a878c70e830ff382eb.

* Update informant log

* Fix string construction

* Refactor resuming snapshots

* Fix informant

* PR Grumbles

* Update informant message : show chunks done

* PR Grumbles

* Fix

* Fix Warning

* PR Grumbles

* Don't open Browser post-install on Mac (#8641)

Since we start parity with the UI disabled per default now, opening the browser post installation will show an annoying error message, confusing the user. This patch removes opening the browser to prevent that annoyance.

fixes #8194

* Fix not downloading old blocks (#8642)

* Fix PoW blockchains sealing notifications in chain_new_blocks (#8656)

* Shutdown the Snapshot Service early (#8658)

* Shutdown the Snapshot Service when shutting down the runner

* Rename `service` to `client_service`

* Fix tests

* Fix cli signer (#8682)

* Update ethereum-types so `{:#x}` applies 0x prefix

* Set the request index to that of the current request (#8683)

* Set the request index to that of the current request

When setting up the chain of (two) requests to look up a block by hash, the second need to refer to the first. This fixes an issue where the back ref was set to the subsequent request, not the current one. When the requests are executed we loop through them in order and ensure the requests that should produce headers all match up. We do this by index so they better be right.

In other words: off by one.

* network-devp2p: handle UselessPeer disconnect (#8686)

* Fix local transactions policy. (#8691)

* CI: Fixes for Android Pipeline (#8745)

* ci: Remove check for shared libraries in gitlab script

* ci: allow android arm build to fail

* Custom Error Messages on ENFILE and EMFILE IO Errors (#8744)

* Custom Error Messages on ENFILE and EMFILE IO Errors

Add custom mapping of ENFILE and EMFILE IO Errors (Failure because of missing system resource) right when chaining ioError into ::util::Network::Error to improve Error Messages given to user

Note: Adds libc as a dependency to util/network

* Use assert-matches for more readable tests

* Fix Wording and consistency

* ethcore-sync: fix connection to peers behind chain fork block (#8710)
2018-06-04 22:29:20 +02:00
Wei Tang
897a94641e Unify and limit rocksdb dependency places (#8371)
* secret_store: remove kvdb_rocksdb dependency

* cli: init db mod for open dispatch

* cli: move db, client_db, restoration_db, secretstore_db to a separate mod

* migration: rename to migration-rocksdb and remove ethcore-migrations

* ethcore: re-move kvdb-rocksdb dep to test

* mark test_helpers as test only and fix migration mod naming

* Move restoration_db_handler to test_helpers_internal

* Fix missing preambles in test_helpers_internal and rocksdb/helpers

* Move test crates downward

* Fix missing docs

* cli, db::open_db: move each argument to a separate line

* Use featuregate instead of dead code for `open_secretstore_db`

* Move pathbuf import to open_secretstore_db

Because it's only used there behind a feature gate
2018-04-13 21:14:53 +02:00
Tomasz Drwięga
1cd93e4ceb New Transaction Queue implementation (#8074)
* Implementation of Verifier, Scoring and Ready.

* Queue in progress.

* TransactionPool.

* Prepare for txpool release.

* Miner refactor [WiP]

* WiP reworking miner.

* Make it compile.

* Add some docs.

* Split blockchain access to a separate file.

* Work on miner API.

* Fix ethcore tests.

* Refactor miner interface for sealing/work packages.

* Implement next nonce.

* RPC compiles.

* Implement couple of missing methdods for RPC.

* Add transaction queue listeners.

* Compiles!

* Clean-up and parallelize.

* Get rid of RefCell in header.

* Revert "Get rid of RefCell in header."

This reverts commit 0f2424c9b7319a786e1565ea2a8a6d801a21b4fb.

* Override Sync requirement.

* Fix status display.

* Unify logging.

* Extract some cheap checks.

* Measurements and optimizations.

* Fix scoring bug, heap size of bug and add cache

* Disable tx queueing and parallel verification.

* Make ethcore and ethcore-miner compile again.

* Make RPC compile again.

* Bunch of txpool tests.

* Migrate transaction queue tests.

* Nonce Cap

* Nonce cap cache and tests.

* Remove stale future transactions from the queue.

* Optimize scoring and write some tests.

* Simple penalization.

* Clean up and support for different scoring algorithms.

* Add CLI parameters for the new queue.

* Remove banning queue.

* Disable debug build.

* Change per_sender limit to be 1% instead of 5%

* Avoid cloning when propagating transactions.

* Remove old todo.

* Post-review fixes.

* Fix miner options default.

* Implement back ready transactions for light client.

* Get rid of from_pending_block

* Pass rejection reason.

* Add more details to drop.

* Rollback heap size of.

* Avoid cloning hashes when propagating and include more details on rejection.

* Fix tests.

* Introduce nonces cache.

* Remove uneccessary hashes allocation.

* Lower the mem limit.

* Re-enable parallel verification.

* Add miner log. Don't check the type if not below min_gas_price.

* Add more traces, fix disabling miner.

* Fix creating pending blocks twice on AuRa authorities.

* Fix tests.

* re-use pending blocks in AuRa

* Use reseal_min_period to prevent too frequent update_sealing.

* Fix log to contain hash not sender.

* Optimize local transactions.

* Fix aura tests.

* Update locks comments.

* Get rid of unsafe Sync impl.

* Review fixes.

* Remove excessive matches.

* Fix compilation errors.

* Use new pool in private transactions.

* Fix private-tx test.

* Fix secret store tests.

* Actually use gas_floor_target

* Fix config tests.

* Fix pool tests.

* Address grumbles.
2018-04-13 17:34:27 +02:00
Anton Gavrilov
e6f75bccfe Private transactions integration pr (#6422)
* Private transaction message added

* Empty line removed

* Private transactions logic removed from client into the separate module

* Fixed compilation after merge with head

* Signed private transaction message added as well

* Comments after the review fixed

* Private tx execution

* Test update

* Renamed some methods

* Fixed some tests

* Reverted submodules

* Fixed build

* Private transaction message added

* Empty line removed

* Private transactions logic removed from client into the separate module

* Fixed compilation after merge with head

* Signed private transaction message added as well

* Comments after the review fixed

* Encrypted private transaction message and signed reply added

* Private tx execution

* Test update

* Main scenario completed

* Merged with the latest head

* Private transactions API

* Comments after review fixed

* Parameters for private transactions added to parity arguments

* New files added

* New API methods added

* Do not process packets from unconfirmed peers

* Merge with ptm_ss branch

* Encryption and permissioning with key server added

* Fixed compilation after merge

* Version of Parity protocol incremented in order to support private transactions

* Doc strings for constants added

* Proper format for doc string added

* fixed some encryptor.rs grumbles

* Private transactions functionality moved to the separate crate

* Refactoring in order to remove late initialisation

* Tests fixed after moving to the separate crate

* Fetch method removed

* Sync test helpers refactored

* Interaction with encryptor refactored

* Contract address retrieving via substate removed

* Sensible gas limit for private transactions implemented

* New private contract with nonces added

* Parsing of the response from key server fixed

* Build fixed after the merge, native contracts removed

* Crate renamed

* Tests moved to the separate directory

* Handling of errors reworked in order to use error chain

* Encodable macro added, new constructor replaced with default

* Native ethabi usage removed

* Couple conversions optimized

* Interactions with client reworked

* Errors omitting removed

* Fix after merge

* Fix after the merge

* private transactions improvements in progress

* private_transactions -> ethcore/private-tx

* making private transactions more idiomatic

* private-tx encryptor uses shared FetchClient and is more idiomatic

* removed redundant tests, moved integration tests to tests/ dir

* fixed failing service test

* reenable add_notify on private tx provider

* removed private_tx tests from sync module

* removed commented out code

* Use plain password instead of unlocking account manager

* remove dead code

* Link to the contract changed

* Transaction signature chain replay protection module created

* Redundant type conversion removed

* Contract address returned by private provider

* Test fixed

* Addressing grumbles in PrivateTransactions (#8249)

* Tiny fixes part 1.

* A bunch of additional comments and todos.

* Fix ethsync tests.

* resolved merge conflicts

* final private tx pr (#8318)

* added cli option that enables private transactions

* fixed failing test

* fixed failing test

* fixed failing test

* fixed failing test
2018-04-09 16:14:33 +02:00
Wei Tang
c039ab79b5 Decouple rocksdb dependency from ethcore (#8320)
* Move client DB opening logic to CLI

* Move restoration db open logic to CLI

This adds KeyValueDBHandler which handles opening a new database, thus allow us to move the restoration db open logic
out of ethcore.

* Move rocksdb's compactionprofile conversion to CLI

* Move kvdb_rocksdb as test dependency for ethcore

* Fix tests due to interface change

* Fix service tests

* Remove unused migration dep for ethcore
2018-04-09 14:21:37 +02:00
Marek Kotewicz
de72643898
ethcore-service (#8089)
* ethcore test::helpers cleanup

* ethcore-service
2018-03-13 11:49:57 +01:00
debris
7849fff41e finished 2017-08-31 11:35:41 +02:00
Tomasz Drwięga
a24b6ad983 Use standard paths for Ethash cache (#5881)
* Use cache path to store ethash files.

* Fixing tests, more flexible API.

* Use AsRef<Path> everywhere.

* Fixing ethcore tests.

* Fix RPC tests.
2017-07-10 12:57:40 +02:00
Arkadiy Paronyan
4d5280e43c removed panic handlers (#5895) 2017-06-22 19:00:53 +02:00
Gav Wood
3041c95408 Chain-selection from UI (#4859)
* First little bits for chain-selection.

* Provide RPCs and get settings through to user defaults.

* Hasty stash.

* Fix updater accidentally redownloading.

* Finish up.

* Add JS tests.

* Hypervisor should never run a binary modified before itself.

* Style.

* Help tweak.

* Fix test compile.

* Fix JS test

* Build fix for tests.

* Revert default chain name

* Another test

* Use spec name via client.

* Fix mock up.

* whitespace

[ci:skip]

* whitespace

[ci:skip]

* remove exit/restart endpoints.
2017-03-13 12:10:53 +01:00
Gav Wood
8404edb656 Fix whitespace (#4299)
* Fix whitespace.

* Update copyright years/owner.

* Push release only for tags.
2017-01-25 18:51:41 +01:00
Robert Habermeier
203fd8a471 Memory-based pruning history size (#4114)
* prune states based on memory param

* pruning memory CLI and usage in sync

* return purged value from memorydb

* calculate memory used incrementally in overlayrecentdb

* refactor shared history pruning code in client

* Fixed usage alignment

* journal_size function for fast memory calculation
2017-01-20 13:25:53 +01:00
Robert Habermeier
8125b5690c Port try macro to new ? operator. (#3962)
* initial untry sweep

* restore try in ipc codegen, fix inference

* change a few missed try instances
2016-12-27 12:53:56 +01:00
arkpar
e2371632c5 Base dir option 2016-12-15 21:56:45 +01:00
arkpar
be5fefe772 Merge branch 'master' of github.com:ethcore/parity into new-dirs 2016-12-14 00:04:11 +01:00
arkpar
6601fde328 New data paths with upgrade 2016-12-12 16:51:07 +01:00
Gav Wood
17cc2be4b5
Rename ethcore -> parity in license. 2016-12-11 19:30:54 +01:00
Gav Wood
5f1fcf95e0
Make *ID names consistent with std Rust (Id) 2016-12-09 23:01:43 +01:00
Robert Habermeier
8c8bf30de5 drop spec when no longer useful 2016-11-15 19:07:23 +01:00
Robert Habermeier
d18bb9d152 verify chunk hashes in cli restore (#3241) 2016-11-07 19:33:55 +01:00
Gav Wood
744501c454 Mode improvements for UI (#3109)
* `--mode=off` now works.

* Add Mode::Off as a persistent CLI option.

* "last" not "auto" as default.

* Commit accidentally unsaved file.

* Whitespace

[ci:skip]

* Mode CLI parse fix

* or offline

* Save mode when it gets changed.

* Fix Offline mode

* Fix up mode operations.

* Make passive default, but not overriding.

* Fix test

* Maybe not everyone wants to run an archive node...
2016-11-05 10:38:00 +01:00
Tomasz Drwięga
cf8f27ce0f Initializing logger for each command (#3090)
* Initializing logger for each command

* Single logger setup

* Whitespace [ci:skip]
2016-11-02 19:42:21 +01:00
Robert Habermeier
bc81ae0407 Snapshot and blockchain stability improvements (#2843)
* allow taking snapshot from just-restored database without error

* make creation informant less spammy

* Ancestry iterator failure-resilient

* make uncle hash searching resilient to incomplete chain

* deduce pre-chunk info from last written block's details
2016-10-24 18:27:23 +02:00
Arkadiy Paronyan
44a560e964 CLI option to skip seal check when importing (#2842) 2016-10-24 15:09:13 +02:00
keorn
479657b23b Automatic compaction selection on Linux (#2785)
* add auto compaction types

* pass db paths

* detect drive type on Linux

* use base db path

* add docstring

* limit the test to be side effect free

* use base db path

* more docs

* fix parsing test

* update error

* detect only on Linux

* make test Linux only

* add second device letter, update cli doc

* use spaces in cli doc

* import only on linux

* default->auto
2016-10-21 23:21:57 +02:00
Arkadiy Paronyan
487dfb0208 Snapshot sync part 2 (#2098)
* Split block downloader into a module

* Snapshot sync progress

* Warp sync CLI option

* Increased snapshot chunk and ping timeouts

* Fixed an issue with delayed writes

* Updated bootnodes

* Don't run pending IO tasks on shutdown

* Optional first_block; removed insert_snapshot_block

* Fixing expect calls

* Fixed stalled sync

* style and docs

* Update block_sync.rs

[ci:skip]
2016-10-18 18:16:00 +02:00
Robert Habermeier
f28b8352c1 Configurable history size in master (#2606)
* split journaldb commit into two functions: journal_under and mark_canonical

* use new commit mechanism in client

* Configurable history size in master

* Reduce DB history

* Configurable history size

* Set min history size

* Test

* Fixed a test and reduced the limit
2016-10-14 15:44:56 +03:00
Marek Kotewicz
06fe768ac2 Polishing Actually enable fat db pr (#1974) (#2048)
* Actually enable fat db, and do RPCs for it.

* Implement HashDB traits for AccountDB.

* user defaults

* finished user defaults

* user defaults are network-dependent

* added tests for newly added functions, logger is initialized first

* dir cleanup in progress

* user_file is placed next to snapshots

* fixing requested change
2016-10-03 11:13:10 +02:00
Marek Kotewicz
56eb97abbf user defaults (#2014)
* user defaults

* finished user defaults

* user defaults are network-dependent

* added tests for newly added functions, logger is initialized first

* dir cleanup in progress

* user_file is placed next to snapshots
2016-09-26 19:21:25 +02:00
Robert Habermeier
dcfd7eab6d Local snapshot restore (#2058)
* restore from local snapshot

* update status with chunks done

* rework local restore trigger
2016-09-11 14:05:59 +02:00
Robert Habermeier
57d5c35bb6 Use proper database configuration in snapshots. (#2052)
* use proper database config in snapshot service

* add snapshot path to parity directories struct

* fix RPC tests
2016-09-07 15:27:28 +02:00
Arkadiy Paronyan
5c5d9c8ccd Snapshot sync (#2047)
* PV64 sync

* Tests

* Client DB restore

* Snapshot restoration over IPC

* Upating test

* Minor tweaks

* Upating test
2016-09-06 15:31:13 +02:00
Robert Habermeier
1c19a807d9 Take control of recovered snapshots, start restoration asynchronously (#2010)
* take control of given snapshot

* start snapshot restoration asynchronously,
2016-08-25 22:20:44 +02:00
Robert Habermeier
b18407b9e3 Snapshot optimizations (#1991)
* apply RLP compression to abridged blocks

* add memorydb consolidate

* code hash optimization

* add warning to snapshot restoration CLI
2016-08-25 14:28:45 +02:00
Nikolay Volf
7c5435d3bb fix to use relative socket path 2016-08-22 19:41:58 +03:00
Tomasz Drwięga
a427208f79 Purging .derefs, fixing clippy warnings. (#1890)
* Fixing clippy warnings

* Purging derefs

* Simplifying engine derefs

* Simplifying more engine derefs
2016-08-10 16:29:40 +02:00
Robert Habermeier
2f1ade8116 take snapshot at specified block and slightly better informants (#1873)
* prettier informant for snapshot creation

* allow taking snapshot at a given block

* minor tweaks

* elaborate on cli
2016-08-08 18:41:30 +02:00
Robert Habermeier
e72fc5398a miner and client take spec reference (#1853)
* miner and client take spec reference

* fix tests
2016-08-05 23:33:55 +02:00
Robert Habermeier
76a7246369 Snapshot creation and restoration (#1679)
* to_rlp takes self by-reference

* clean up some derefs

* out-of-order insertion for blockchain

* implement block rebuilder without verification

* group block chunk header into struct

* block rebuilder does verification

* integrate snapshot service with client service; flesh out implementation more

* initial implementation of snapshot service

* remove snapshottaker trait

* snapshot writer trait with packed and loose implementations

* write chunks using "snapshotwriter" in service

* have snapshot taking use snapshotwriter

* implement snapshot readers

* back up client dbs when replacing

* use snapshot reader in snapshot service

* describe offset format

* use new get_db_path in parity, allow some errors in service

* blockchain formatting

* implement parity snapshot

* implement snapshot restore

* force blocks to be submitted in order

* fix bug loading block hashes in packed reader

* fix seal field loading

* fix uncle hash computation

* fix a few bugs

* store genesis state in db. reverse block chunk order in packed writer

* allow out-of-order import for blocks

* bring restoration types together

* only snapshot the last 30000 blocks

* restore into overlaydb instead of journaldb

* commit version to database

* use memorydbs and commit directly

* fix trie test compilation

* fix failing tests

* sha3_null_rlp, not H256::zero

* move overlaydb to ref_overlaydb, add new overlaydb without on-disk rc

* port archivedb to new overlaydb

* add deletion mode tests for overlaydb

* use new overlaydb, check state root at end

* share chain info between state and block snapshotting

* create blocks snapshot using blockchain directly

* allow snapshot from arbitrary block, remove panickers from snapshot creation

* begin test framework

* blockchain chunking test

* implement stateproducer::tick

* state snapshot test

* create block and state chunks concurrently, better restoration informant

* fix tests

* add deletion mode tests for overlaydb

* address comments

* more tests

* Fix up tests.

* remove a few printlns

* add a little more documentation to `commit`

* fix tests

* fix ref_overlaydb test names

* snapshot command skeleton

* revert ref_overlaydb renaming

* reimplement snapshot commands

* fix many errors

* everything but inject

* get ethcore compiling

* get snapshot tests passing again

* instrument snapshot commands again

* fix fallout from other changes, mark snapshots as experimental

* optimize injection patterns

* do two injections

* fix up tests

* take snapshots from 1000 blocks efore

* address minor comments

* fix a few io crate related errors

* clarify names about total difficulty

[ci skip]
2016-08-05 17:00:46 +02:00