Commit Graph

101 Commits

Author SHA1 Message Date
Andrew Jones
0b34579b04
Prevent sync restart if import queue full (#9381) 2018-08-24 09:42:24 +01:00
Tomasz Drwięga
29125e830b Lower the max size of transaction packet to prevent going oversize. (#9308)
* Lower the max size of transaction packet to prevent going oversize.

* Log RLP size.
2018-08-14 17:20:29 +02:00
Andronik Ordian
98dbd1fdc7
Fix load share (#9321)
* fix(light_sync): calculate `load_share` properly

* refactor(api.rs): extract `light_params` fn, add test

* style(api.rs): add trailing commas
2018-08-13 18:53:19 +03:00
Niklas Adolfsson
6e2821b4db ethcore/sync Make view macro only visible to test (#9316)
* remove needless macro import

* enable ethcore/macros in tests
2018-08-10 14:31:48 +02:00
Nick Sanders
e590874a81 Update log -> 0.4, env_logger -> 0.5. (#9294)
* Rename a few types & methods.

* Change `(Log)Builder::format` (closure) arg.
2018-08-10 00:04:10 +01:00
David
e2095d4a5d Move ethereum-specific H256FastMap type to own crate (#9307)
* Add a `fastmap` crate that provides the H256FastMap specialized HashMap

* Use `fastmap` instead of `plain_hasher`

* Update submodules for Reasons™

* Submodule update
2018-08-09 09:51:48 +02:00
Marek Kotewicz
78a38e9825
ethcore sync decodes rlp less often (#9264)
* deserialize block only once during verification

* ethcore-sync uses Unverified

* ethcore-sync uses Unverified

* fixed build error

* removed Block::is_good

* applied review suggestions

* ethcore-sync deserializes headers and blocks only once
2018-08-08 10:56:54 +02:00
Tomasz Drwięga
90d7823acb Propagate transactions for next 4 blocks. (#9265)
Closes #9255 

This PR also removes the limit of max 64 transactions per packet, currently we only attempt to prevent the packet size to go over 8MB. This will only be the case for super-large transactions or high-block-gas-limit chains.

Patching this is important only for chains that have blocks that can fit more than 4k transactions (over 86M block gas limit)

For mainnet, we should actually see a tiny bit faster propagation since instead of computing 4k pending set, we only need `4 * 8M / 21k = 1523` transactions.

Running some tests on `dekompile` node right now, to check how it performs in the wild.
2018-08-02 11:58:02 +01:00
Marek Kotewicz
b4ae1b6528 decode block rlp less often (#9252)
in total:
- removed 4 redundant rlp deserializations
- avoid 1 redundant block data copy
2018-08-02 10:20:46 +01:00
Andrew Jones
fb503f523b
Allow old blocks from peers with lower difficulty (#9226)
Previously we only allow downloading of old blocks if the peer
difficulty was greater than our syncing difficulty. This change allows
downloading of blocks from peers where the difficulty is greater then
the last downloaded old block.
2018-07-27 10:13:05 +01:00
Marek Kotewicz
073365d5d9
handle SyncHandler errors properly (#9151)
* handle SyncHandler errors properly, closes #9150

* applied review suggestions
2018-07-19 12:46:33 +02:00
Marek Kotewicz
0ca4250bd4 fix verification in ethcore-sync collect_blocks (#9135) 2018-07-16 21:05:47 +02:00
Afri Schoedon
484ecfaf47
Parity Ethereum 2.0.0 (#9052)
* parity-version: major bump to 2.0.0 🎉

* parity-ethereum: rename crate 🌵

* ethcore: only accept service transactions from parity-ethereum nodes

* parity: fix --identity tests

* rpc: fix sync provider in tests

* rpc: fix parity_net_peers test

* ethcore-sync: accept service transactions from parity and parity-ethereum

* ethcore-sync: fix indentation

* ethcore-sync: split the ifs to reduce code redundancy

* ethcore-sync: fix syntax

* Fix building ethcore

* update cargo.lock

* parity-version: major bump to 2.0.0 tada

* fix merge
2018-07-11 13:35:10 +02:00
Nicolas Gotchac
c082af6f74 Don't fetch snapshot chunks at random (#9088) 2018-07-11 13:04:31 +02:00
David
c7f608ec74
Delete crates from parity-ethereum and fetch them from parity-common instead (#9083)
Use crates from parity-common: hashdb, keccak-hash, kvdb, kvdb-memorydb, kvdb-rocksdb, memorydb, parity-bytes, parity-crypto, path, patricia_trie, plain_hasher, rlp, target, test-support, trie-standardmap, triehash
2018-07-10 14:59:19 +02:00
Tomasz Drwięga
aa67bd5d00 A last bunch of txqueue performance optimizations (#9024)
* Clear cache only when block is enacted.

* Add tracing for cull.

* Cull split.

* Cull after creating pending block.

* Add constant, remove sync::read tracing.

* Reset debug.

* Remove excessive tracing.

* Use struct for NonceCache.

* Fix build

* Remove warnings.

* Fix build again.
2018-07-05 16:27:48 +01:00
Wei Tang
e3e2fcc285 bump parking_lot to 0.6 (#9013)
* Bump parking_lot to 0.6

* Bump parity-wasm to 0.31 so it gets rid of parking_lot

ref https://github.com/paritytech/parity-wasm/pull/206

* Update jsonrpc versions

* Update wasmi and pwasm-utils version

* Fix compile

* Update jsonrpc crates
2018-07-03 17:31:08 +02:00
Tomasz Drwięga
78e001284f Recently rejected cache for transaction queue (#9005)
* Store recently rejected transactions.

* Don't cache AlreadyImported rejections.

* Make the size of transaction verification queue dependent on pool size.

* Add a test for recently rejected.

* Fix logging for recently rejected.

* Make rejection cache smaller.

* obsolete test removed

* obsolete test removed

* Construct cache with_capacity.
2018-07-02 18:00:06 +01:00
David
9caa868603
Make HashDB generic (#8739)
The `patricia_trie` crate is generic over the hasher (by way of HashDB) and node encoding scheme. Adds a new `patricia_trie_ethereum` crate with concrete impls for Keccak/RLP.
2018-07-02 18:50:05 +02:00
Toralf Wittner
41348dead4 Add type for passwords. (#8920)
* Add type for passwords.

* Fix test.

* Simplify `Drop` impls of `Password` and `Memzero`.

* Spaces to tabs.

* Custom `Drop` impl for `Password`.
2018-06-22 15:09:15 +02:00
Marek Kotewicz
458afcd230 new blooms database (#8712)
* new blooms database

* fixed conflict in Cargo.lock

* removed bloomchain

* cleanup in progress

* all tests passing in trace db with new blooms-db

* added trace_blooms to BlockChainDB interface, fixed db flushing

* BlockChainDB no longer exposes RwLock in the interface

* automatically flush blooms-db after every insert

* blooms-db uses io::BufReader to read files, wrap blooms-db into Mutex, cause fs::File is just a shared file handle

* fix json_tests

* blooms-db can filter multiple possibilities at the same time

* removed enum trace/db.rs CacheId

* lint fixes

* fixed tests

* kvdb-rocksdb uses fs-swap crate

* update Cargo.lock

* use fs::rename

* fixed failing test on linux

* fix tests

* use fs_swap

* fixed failing test on linux

* cleanup after swap

* fix tests

* fixed osx permissions

* simplify parity database opening functions

* added migration to blooms-db

* address @niklasad1 grumbles

* fix license and authors field of blooms-db Cargo.toml

* restore blooms-db after snapshot
2018-06-20 15:13:07 +02:00
André Silva
3016d54f13 Minor fix in chain supplier and light provider (#8906)
* fix chain supplier increment

* fix light provider block_headers
2018-06-15 17:30:34 +02:00
Tomasz Drwięga
4938d5dde5 Limit the number of transactions in pending set (#8777)
* Unordered iterator.

* Use unordered and limited set if full not required.

* Split timeout work into smaller timers.

* Avoid collecting all pending transactions when mining

* Remove println.

* Use priority ordering in eth-filter.

* Fix ethcore-miner tests and tx propagation.

* Review grumbles addressed.

* Add test for unordered not populating the cache.

* Fix ethcore tests.

* Fix light tests.

* Fix ethcore-sync tests.

* Fix RPC tests.
2018-06-12 08:22:54 +02:00
Wei Tang
e2a90ce159 Conditionally compile ethcore public test helpers (#8743)
* Mark test helpers and test-only specs as cfg(test)

* Use test-probe to conditionally compile test helpers

* Remove test probe and directly use features tag
2018-06-04 15:58:44 +02:00
Niklas Adolfsson
98b7c07171 Update license header and scripts (#8666)
* Update `add_license` script

* run script

* add `remove duplicate lines script` and run it

* Revert changes `English spaces`

* strip whitespaces

* Revert `GPL` in files with `apache/mit license`

* don't append `gpl license` in files with other lic

* Don't append `gpl header` in files with other lic.

* re-ran script

* include c and cpp files too

* remove duplicate header

* rebase nit
2018-06-04 10:19:50 +02:00
Pierre Krieger
3d76417353 Remove HostTrait altogether (#8681) 2018-06-02 11:05:11 +02:00
André Silva
2060ea5de3 ethcore-sync: fix connection to peers behind chain fork block (#8710) 2018-06-02 10:29:44 +02:00
Pierre Krieger
dab967ace8 Remove NetworkService::config() (#8653) 2018-06-01 09:49:46 +02:00
Nicolas Gotchac
7fcb082cad Shutdown the Snapshot Service early (#8658)
* Shutdown the Snapshot Service when shutting down the runner

* Rename `service` to `client_service`

* Fix tests
2018-05-29 13:23:15 +03:00
Pierre Krieger
fe5f5b28d9 Remove the error when stopping the network (#8671) 2018-05-22 12:35:13 +08:00
Pierre Krieger
ee41fa6f30 Move connection_filter to the network crate (#8674) 2018-05-22 12:34:01 +08:00
Nicolas Gotchac
d1934363e7 Fix not downloading old blocks (#8642) 2018-05-17 10:58:35 +02:00
Nicolas Gotchac
cdbcfaa7de Resumable warp-sync / Seed downloaded snapshots (#8544)
* Start dividing sync chain : first supplier method

* WIP - updated chain sync supplier

* Finish refactoring the Chain Sync Supplier

* Create Chain Sync Requester

* Add Propagator for Chain Sync

* Add the Chain Sync Handler

* Move tests from mod -> handler

* Move tests to propagator

* Refactor SyncRequester arguments

* Refactoring peer fork header handler

* Fix wrong highest block number in snapshot sync

* Small refactor...

* Resume warp-sync downloaded chunks

* Add comments

* Refactoring the previous chunks import

* Fix tests

* Address PR grumbles

* Fix not seeding current snapshot

* Address PR Grumbles

* Address PR grumble

* Retry failed CI job

* Update SnapshotService readiness check
Fix restoration locking issue for previous chunks restoration

* Fix tests

* Fix tests

* Fix test

* Early abort importing previous chunks

* PR Grumbles

* Update Gitlab CI config

* SyncState back to Waiting when Manifest peers disconnect

* Move fix

* Better fix

* Revert GitLab CI changes

* Fix Warning

* Refactor resuming snapshots

* Fix string construction

* Revert "Refactor resuming snapshots"

This reverts commit 75fd4b553a38e4a49dc5d6a878c70e830ff382eb.

* Update informant log

* Fix string construction

* Refactor resuming snapshots

* Fix informant

* PR Grumbles

* Update informant message : show chunks done

* PR Grumbles

* Fix

* Fix Warning

* PR Grumbles
2018-05-16 22:01:55 +02:00
Tomasz Drwięga
08abf67a51 Fix packet count when talking with PAR2 peers (#8555)
* Support diferent packet counts in different protocol versions.

* Fix light timeouts and eclipse protection.

* Fix devp2p tests.

* Fix whisper-cli compilation.

* Fix compilation.

* Fix ethcore-sync tests.

* Revert "Fix light timeouts and eclipse protection."

This reverts commit 06285ea8c1d9d184d809f64b5507aece633da6cc.

* Increase timeouts.
2018-05-14 10:09:05 +02:00
Niklas Adolfsson
ecd7caa93d fix compiler warning (#8590) 2018-05-11 12:23:19 +03:00
Pierre Krieger
1b8f299df2 Make mio optional in ethcore-io (#8537)
* Make mio optional in ethcore-io

* Add some annotations, plus a check for features

* Increase timer for test
2018-05-10 12:34:36 +02:00
David
842b75c0e6 Decoding headers can fail (#8570)
* rlp::decode returns Result

* Fix journaldb to handle rlp::decode Result

* Fix ethcore to work with rlp::decode returning Result

* Light client handles rlp::decode returning Result

* Fix tests in rlp_derive

* Fix tests

* Cleanup

* cleanup

* Allow panic rather than breaking out of iterator

* Let decoding failures when reading from disk blow up

* syntax

* Fix the trivial grumbles

* Fix failing tests

* Make Account::from_rlp return Result

* Syntx, sigh

* Temp-fix for decoding failures

* Header::decode returns Result

Handle new return type throughout the code base.

* Do not continue reading from the DB when a value could not be read

* Fix tests

* Handle header decoding in light_sync

* Handling header decoding errors

* Let the DecodeError bubble up unchanged

* Remove redundant error conversion
2018-05-09 12:05:56 +02:00
Nicolas Gotchac
8b0ba97cf2 Refactoring ethcore-sync - Fixing warp-sync barrier (#8543)
* Start dividing sync chain : first supplier method

* WIP - updated chain sync supplier

* Finish refactoring the Chain Sync Supplier

* Create Chain Sync Requester

* Add Propagator for Chain Sync

* Add the Chain Sync Handler

* Move tests from mod -> handler

* Move tests to propagator

* Refactor SyncRequester arguments

* Refactoring peer fork header handler

* Fix wrong highest block number in snapshot sync

* Small refactor...

* Address PR grumbles

* Retry failed CI job

* Fix tests

* PR Grumbles
2018-05-09 12:05:34 +02:00
Tomasz Drwięga
24838bbcd3 Don't block sync when importing old blocks (#8530)
* Alter IO queueing.

* Don't require IoMessages to be Clone

* Ancient blocks imported via IoChannel.

* Get rid of private transactions io message.

* Get rid of deadlock and fix disconnected handler.

* Revert to old disconnect condition.

* Fix tests.

* Fix deadlock.
2018-05-09 08:49:34 +02:00
Wei Tang
528497b86a Keep all enacted blocks notify in order (#8524)
* Keep all enacted blocks notify in order

* Collect is unnecessary

* Update ChainNotify to use ChainRouteType

* Fix all ethcore fn defs

* Wrap the type within ChainRoute

* Fix private-tx and sync api

* Fix secret_store API

* Fix updater API

* Fix rpc api

* Fix informant api

* Eagerly cache enacted/retracted and remove contain_enacted/retracted

* Fix indent

* tests: should use full expr form for struct constructor

* Use into_enacted_retracted to further avoid copy

* typo: not a function

* rpc/tests: ChainRoute -> ChainRoute::new
2018-05-07 12:58:25 +02:00
Niklas Adolfsson
e36c4ecc98 duration_ns: u64 -> duration: Duration (#8457)
* duration_ns: u64 -> duration: Duration

* format on millis {:.2} -> {}
2018-04-27 15:04:27 +02:00
Anton Gavrilov
01d399ad66 Return error if RLP size of transaction exceeds the limit (#8473)
* Return error if RLP size of transaction exceeds the limit

* Review comments fixed

* RLP check moved to verifier, corresponding pool test added
2018-04-27 15:02:45 +02:00
Afri Schoedon
adc3457a89
Bump master to 1.12 (#8477)
* Bump master to 1.12

* Bump crates to 1.12

* Bump mac installer version to 1.12

* Update Gitlab scripts
2018-04-25 16:25:43 +02:00
Tomasz Drwięga
fa261ebd02 Don't require write lock when fetching status. (#8481) 2018-04-25 15:39:45 +02:00
Anton Gavrilov
d86333af6b Private transactions processing error handling (#8431)
* Integration test for private transaction returned

* Do not interrupt verification in case of errors

* Helpers use specified

* Review comments fixed
2018-04-20 15:45:53 +02:00
Andrew Jones
14361cc7b1 Move ethcore::Error to error_chain (#8386)
* WIP

* Convert Ethcore error to use error_chain

* Use error_chain for ImportError and BlockImportError

* Fix error pattern matches for error_chain in miner

* Implement explicit From for AccountsError

* Fix pattern matches for ErrorKinds

* Handle ethcore error_chain in light client

* Explicitly define Result type to avoid shadowing

* Fix remaining Error pattern matches

* Fix tab space formatting

* Helps if the tests compile

* Fix error chain matching after merge
2018-04-19 11:52:54 +02:00
Andrew Jones
a04c5b180a Replace legacy Rlp with UntrustedRlp and use in ethcore rlp views (#8316)
* WIP

* Replace Rlp with UntrustedRlp in views, explicity unwrap with expect

First pass to get it to compile. Need to figure out whether to do this or to propogate Errors upstream, which would require many more changes to dependent code. If we do this way we are assuming that the views are always used in a context where the rlp is trusted to be valid e.g. when reading from our own DB. So need to fid out whether views are used with data received from an untrusted (e.g. extrernal peer).

* Remove original Rlp impl, rename UntrustedRlp -> Rlp

* Create rlp views with view! macro to record debug info

Views are assumed to be over valid rlp, so if there is a decoding error we record where the view was created in the first place and report it in the expect

* Use $crate in view! macro to avoid import, fix tests

* Expect valid rlp in decode functions for now

* Replace spaces with tabs in new file

* Add doc tests for creating views with macro

* Update rlp docs to reflect removing of UntrustedRlp

* Replace UntrustedRlp usages in private-tx merge
2018-04-16 15:52:12 +02:00
Pierre Krieger
fac356c701 More code refactoring to integrate Duration (#8322)
* More code refactoring to integrate Duration

* Fix typo

* Fix tests

* More test fix
2018-04-14 21:35:58 +02:00
Tomasz Drwięga
1cd93e4ceb New Transaction Queue implementation (#8074)
* Implementation of Verifier, Scoring and Ready.

* Queue in progress.

* TransactionPool.

* Prepare for txpool release.

* Miner refactor [WiP]

* WiP reworking miner.

* Make it compile.

* Add some docs.

* Split blockchain access to a separate file.

* Work on miner API.

* Fix ethcore tests.

* Refactor miner interface for sealing/work packages.

* Implement next nonce.

* RPC compiles.

* Implement couple of missing methdods for RPC.

* Add transaction queue listeners.

* Compiles!

* Clean-up and parallelize.

* Get rid of RefCell in header.

* Revert "Get rid of RefCell in header."

This reverts commit 0f2424c9b7319a786e1565ea2a8a6d801a21b4fb.

* Override Sync requirement.

* Fix status display.

* Unify logging.

* Extract some cheap checks.

* Measurements and optimizations.

* Fix scoring bug, heap size of bug and add cache

* Disable tx queueing and parallel verification.

* Make ethcore and ethcore-miner compile again.

* Make RPC compile again.

* Bunch of txpool tests.

* Migrate transaction queue tests.

* Nonce Cap

* Nonce cap cache and tests.

* Remove stale future transactions from the queue.

* Optimize scoring and write some tests.

* Simple penalization.

* Clean up and support for different scoring algorithms.

* Add CLI parameters for the new queue.

* Remove banning queue.

* Disable debug build.

* Change per_sender limit to be 1% instead of 5%

* Avoid cloning when propagating transactions.

* Remove old todo.

* Post-review fixes.

* Fix miner options default.

* Implement back ready transactions for light client.

* Get rid of from_pending_block

* Pass rejection reason.

* Add more details to drop.

* Rollback heap size of.

* Avoid cloning hashes when propagating and include more details on rejection.

* Fix tests.

* Introduce nonces cache.

* Remove uneccessary hashes allocation.

* Lower the mem limit.

* Re-enable parallel verification.

* Add miner log. Don't check the type if not below min_gas_price.

* Add more traces, fix disabling miner.

* Fix creating pending blocks twice on AuRa authorities.

* Fix tests.

* re-use pending blocks in AuRa

* Use reseal_min_period to prevent too frequent update_sealing.

* Fix log to contain hash not sender.

* Optimize local transactions.

* Fix aura tests.

* Update locks comments.

* Get rid of unsafe Sync impl.

* Review fixes.

* Remove excessive matches.

* Fix compilation errors.

* Use new pool in private transactions.

* Fix private-tx test.

* Fix secret store tests.

* Actually use gas_floor_target

* Fix config tests.

* Fix pool tests.

* Address grumbles.
2018-04-13 17:34:27 +02:00
Robert Habermeier
869fa6fda8
handle queue import errors a bit more gracefully (#8385) 2018-04-13 14:06:22 +02:00
Marek Kotewicz
86446d713a ethcore-sync (#8347) 2018-04-10 12:13:49 +02:00