Commit Graph

28 Commits

Author SHA1 Message Date
Tomasz Drwięga
0b5bbf6048 Improve block and transaction propagation (#9954)
* Refactor sync to add priority tasks.

* Send priority tasks notifications.

* Propagate blocks, optimize transactions.

* Implement transaction propagation. Use sync_channel.

* Tone down info.

* Prevent deadlock by not waiting forever for sync lock.

* Fix lock order.

* Don't use sync_channel to prevent deadlocks.

* Fix tests.
2018-11-28 11:30:05 +01:00
Nicolas Gotchac
9475a2e474 Keep existing blocks when restoring a Snapshot (#8643)
* Rename db_restore => client

* First step: make it compile!

* Second step: working implementation!

* Refactoring

* Fix tests

* PR Grumbles

* PR Grumbles WIP

* Migrate ancient blocks interating backward

* Early return in block migration if snapshot is aborted

* Remove RwLock getter (PR Grumble I)

* Remove dependency on `Client`: only used Traits

* Add test for recovering aborted snapshot recovery

* Add test for migrating old blocks

* Fix build

* PR Grumble I

* PR Grumble II

* PR Grumble III

* PR Grumble IV

* PR Grumble V

* PR Grumble VI

* Fix one test

* Fix test

* PR Grumble

* PR Grumbles

* PR Grumbles II

* Fix tests

* Release RwLock earlier

* Revert Cargo.lock

* Update _update ancient block_ logic: set local in `commit`

* Update typo in ethcore/src/snapshot/service.rs

Co-Authored-By: ngotchac <ngotchac@gmail.com>
2018-11-18 00:06:34 +01:00
David
c313039526 Update a few parity-common dependencies (#9663)
* Update a few parity-common dependencies

* cleanup

* cleanup

* revert update of ethereum/tests

* better reporting of network rlp errors

* Use rlp 0.3.0-beta.1

* fix util function get_dummy_blocks

* Already a Vec

* encode_list returns vec already

* Address grumble

* No need for betas

* Fix double spaces
2018-10-09 22:07:25 +02:00
Andrew Jones
4b6ebcbb61 Fix ancient blocks sync (#9531)
* Log block set in block_sync for easier debugging

* logging macros

* Match no args in sync logging macros

* Add QueueFull error

* Only allow importing headers if the first matches requested

* WIP

* Test for chain head gaps and log

* Calc distance even with 2 heads

* Revert previous commits, preparing simple fix

This reverts commit 5f38aa885b22ebb0e3a1d60120cea69f9f322628.

* Reject headers with no gaps when ChainHead

* Reset block sync download when queue full

* Simplify check for subchain heads

* Add comment to explain subchain heads filter

* Fix is_subchain_heads check and comment

* Prevent premature round completion after restart

This is a problem on mainnet where multiple stale peer requests will
force many rounds to complete quickly, forcing the retraction.

* Reset stale old blocks request after queue full

* Revert "Reject headers with no gaps when ChainHead"

This reverts commit 0eb865539e5dee37ab34f168f5fb643300de5ace.

* Add BlockSet to BlockDownloader logging

Currently it is difficult to debug this because there are two instances,
one for OldBlocks and one for NewBlocks. This adds the BlockSet to all
log messages for easy log filtering.

* Reset OldBlocks download from last enqueued

Previously when the ancient block queue was full it would restart the
download from the last imported block, so the ones still in the queue would be
redownloaded. Keeping the existing downloader instance and just
resetting it will start again from the last enqueued block.:wq

* Ignore expired Body and Receipt requests

* Log when ancient block download being restarted

* Only request old blocks from peers with >= difficulty

https://github.com/paritytech/parity-ethereum/pull/9226 might be too
permissive and causing the behaviour of the retraction soon after the
fork block. With this change the peer difficulty has to be greater than
or euqal to our syncing difficulty, so should still fix
https://github.com/paritytech/parity-ethereum/issues/9225

* Some logging and clear stalled blocks head

* Revert "Some logging and clear stalled blocks head"

This reverts commit 757641d9b817ae8b63fec684759b0815af9c4d0e.

* Reset stalled header if useless more than once

* Store useless headers in HashSet

* Add sync target to logging macro

* Don't disable useless peer and fix log macro

* Clear useless headers on reset and comments

* Use custom error for collecting blocks

Previously we resued BlockImportError, however only the Invalid case and
this made little sense with the QueueFull error.

* Remove blank line

* Test for reset sync after consecutive useless headers

* Don't reset after consecutive headers when chain head

* Delete commented out imports

* Return DownloadAction from collect_blocks instead of error

* Don't reset after round complete, was causing test hangs

* Add comment explaining reset after useless

* Replace HashSet with counter for useless headers

* Refactor sync reset on bad block/queue full

* Add missing target for log message

* Fix compiler errors and test after merge

* ethcore: revert ethereum tests submodule update
2018-10-09 15:31:40 +02:00
André Silva
dc14cce7a9 ethcore: fix detection of major import (#9552)
* sync: set state to idle after sync is completed

* sync: refactor sync reset
2018-10-08 12:04:38 +01:00
Jim Posen
2fc1679886 Verify block syncing responses against requests (#9670)
* sync: Validate received BlockHeaders packets against stored request.

* sync: Validate received BlockBodies and BlockReceipts.

* sync: Fix broken tests.

* sync: Unit tests for BlockDownloader::import_headers.

* sync: Unit tests for import_{bodies,receipts}.

* tests: Add missing method doc.
2018-10-03 18:35:10 +08:00
Tomasz Drwięga
cc963d42a0 RPC: parity_getBlockReceipts (#9527)
* Block receipts RPC.

* Use lazy evaluation of block receipts (ecrecover).

* Optimize transaction_receipt to prevent performance regression.

* Fix RPC grumbles.

* Add block & transaction receipt tests.

* Fix conversion to block id.
2018-09-25 18:06:14 +01:00
Marek Kotewicz
b57607e7d3
simplify ethcore errors by removing BlockImportError (#9593) 2018-09-24 11:28:54 +01:00
Nicolas Gotchac
753fd4bda3 Improve P2P discovery (#9526)
* Add `target` to Rust traces

* network-devp2p: Don't remove discovery peer in main sync

* network-p2p: Refresh discovery more often

* Update Peer discovery protocol

* Run discovery more often when not enough nodes connected

* Start the first discovery early

* Update fast discovery rate

* Fix tests

* Fix `ping` tests

* Fixing remote Node address ; adding PingPong round

* Fix tests: update new +1 PingPong round

* Increase slow Discovery rate
Check in flight FindNode before pings

* Add `deprecated` to deprecated_echo_hash

* Refactor `discovery_round` branching
2018-09-14 22:18:03 +02:00
Andrew Jones
bc53e78a04
Remove unused BlockStatus::Pending (#9447)
Pending case never instantiated, and only ever matched together with Unknown
2018-08-31 12:13:01 +01:00
Anton Gavrilov
1073d56245 Private packets verification and queue refactoring (#8715)
* Verify private transaction before propagating

* Private transactions queue reworked with tx pool queue direct usage

* Styling fixed

* Prevent resending private packets to the sender

* Process signed private transaction packets via io queue

* Test fixed

* Build and test fixed after merge

* Comments after review fixed

* Signed transaction taken from verified

* Fix after merge

* Pool scoring generalized in order to use externally

* Lib refactored according to the review comments

* Ready state refactored

* Redundant bound and copying removed

* Fixed build after the merge

* Forgotten case reworked

* Review comments fixed

* Logging reworked, target added

* Fix after merge
2018-08-29 20:31:04 +08:00
Marek Kotewicz
b87c7cac54 block view! removal in progress (#9397) 2018-08-24 10:53:31 +01:00
Tomasz Drwięga
29125e830b Lower the max size of transaction packet to prevent going oversize. (#9308)
* Lower the max size of transaction packet to prevent going oversize.

* Log RLP size.
2018-08-14 17:20:29 +02:00
David
e2095d4a5d Move ethereum-specific H256FastMap type to own crate (#9307)
* Add a `fastmap` crate that provides the H256FastMap specialized HashMap

* Use `fastmap` instead of `plain_hasher`

* Update submodules for Reasons™

* Submodule update
2018-08-09 09:51:48 +02:00
Tomasz Drwięga
90d7823acb Propagate transactions for next 4 blocks. (#9265)
Closes #9255 

This PR also removes the limit of max 64 transactions per packet, currently we only attempt to prevent the packet size to go over 8MB. This will only be the case for super-large transactions or high-block-gas-limit chains.

Patching this is important only for chains that have blocks that can fit more than 4k transactions (over 86M block gas limit)

For mainnet, we should actually see a tiny bit faster propagation since instead of computing 4k pending set, we only need `4 * 8M / 21k = 1523` transactions.

Running some tests on `dekompile` node right now, to check how it performs in the wild.
2018-08-02 11:58:02 +01:00
Marek Kotewicz
b4ae1b6528 decode block rlp less often (#9252)
in total:
- removed 4 redundant rlp deserializations
- avoid 1 redundant block data copy
2018-08-02 10:20:46 +01:00
Andrew Jones
fb503f523b
Allow old blocks from peers with lower difficulty (#9226)
Previously we only allow downloading of old blocks if the peer
difficulty was greater than our syncing difficulty. This change allows
downloading of blocks from peers where the difficulty is greater then
the last downloaded old block.
2018-07-27 10:13:05 +01:00
Marek Kotewicz
073365d5d9
handle SyncHandler errors properly (#9151)
* handle SyncHandler errors properly, closes #9150

* applied review suggestions
2018-07-19 12:46:33 +02:00
Afri Schoedon
484ecfaf47
Parity Ethereum 2.0.0 (#9052)
* parity-version: major bump to 2.0.0 🎉

* parity-ethereum: rename crate 🌵

* ethcore: only accept service transactions from parity-ethereum nodes

* parity: fix --identity tests

* rpc: fix sync provider in tests

* rpc: fix parity_net_peers test

* ethcore-sync: accept service transactions from parity and parity-ethereum

* ethcore-sync: fix indentation

* ethcore-sync: split the ifs to reduce code redundancy

* ethcore-sync: fix syntax

* Fix building ethcore

* update cargo.lock

* parity-version: major bump to 2.0.0 tada

* fix merge
2018-07-11 13:35:10 +02:00
Tomasz Drwięga
78e001284f Recently rejected cache for transaction queue (#9005)
* Store recently rejected transactions.

* Don't cache AlreadyImported rejections.

* Make the size of transaction verification queue dependent on pool size.

* Add a test for recently rejected.

* Fix logging for recently rejected.

* Make rejection cache smaller.

* obsolete test removed

* obsolete test removed

* Construct cache with_capacity.
2018-07-02 18:00:06 +01:00
David
9caa868603
Make HashDB generic (#8739)
The `patricia_trie` crate is generic over the hasher (by way of HashDB) and node encoding scheme. Adds a new `patricia_trie_ethereum` crate with concrete impls for Keccak/RLP.
2018-07-02 18:50:05 +02:00
André Silva
3016d54f13 Minor fix in chain supplier and light provider (#8906)
* fix chain supplier increment

* fix light provider block_headers
2018-06-15 17:30:34 +02:00
Tomasz Drwięga
4938d5dde5 Limit the number of transactions in pending set (#8777)
* Unordered iterator.

* Use unordered and limited set if full not required.

* Split timeout work into smaller timers.

* Avoid collecting all pending transactions when mining

* Remove println.

* Use priority ordering in eth-filter.

* Fix ethcore-miner tests and tx propagation.

* Review grumbles addressed.

* Add test for unordered not populating the cache.

* Fix ethcore tests.

* Fix light tests.

* Fix ethcore-sync tests.

* Fix RPC tests.
2018-06-12 08:22:54 +02:00
Niklas Adolfsson
98b7c07171 Update license header and scripts (#8666)
* Update `add_license` script

* run script

* add `remove duplicate lines script` and run it

* Revert changes `English spaces`

* strip whitespaces

* Revert `GPL` in files with `apache/mit license`

* don't append `gpl license` in files with other lic

* Don't append `gpl header` in files with other lic.

* re-ran script

* include c and cpp files too

* remove duplicate header

* rebase nit
2018-06-04 10:19:50 +02:00
André Silva
2060ea5de3 ethcore-sync: fix connection to peers behind chain fork block (#8710) 2018-06-02 10:29:44 +02:00
Nicolas Gotchac
cdbcfaa7de Resumable warp-sync / Seed downloaded snapshots (#8544)
* Start dividing sync chain : first supplier method

* WIP - updated chain sync supplier

* Finish refactoring the Chain Sync Supplier

* Create Chain Sync Requester

* Add Propagator for Chain Sync

* Add the Chain Sync Handler

* Move tests from mod -> handler

* Move tests to propagator

* Refactor SyncRequester arguments

* Refactoring peer fork header handler

* Fix wrong highest block number in snapshot sync

* Small refactor...

* Resume warp-sync downloaded chunks

* Add comments

* Refactoring the previous chunks import

* Fix tests

* Address PR grumbles

* Fix not seeding current snapshot

* Address PR Grumbles

* Address PR grumble

* Retry failed CI job

* Update SnapshotService readiness check
Fix restoration locking issue for previous chunks restoration

* Fix tests

* Fix tests

* Fix test

* Early abort importing previous chunks

* PR Grumbles

* Update Gitlab CI config

* SyncState back to Waiting when Manifest peers disconnect

* Move fix

* Better fix

* Revert GitLab CI changes

* Fix Warning

* Refactor resuming snapshots

* Fix string construction

* Revert "Refactor resuming snapshots"

This reverts commit 75fd4b553a38e4a49dc5d6a878c70e830ff382eb.

* Update informant log

* Fix string construction

* Refactor resuming snapshots

* Fix informant

* PR Grumbles

* Update informant message : show chunks done

* PR Grumbles

* Fix

* Fix Warning

* PR Grumbles
2018-05-16 22:01:55 +02:00
Tomasz Drwięga
08abf67a51 Fix packet count when talking with PAR2 peers (#8555)
* Support diferent packet counts in different protocol versions.

* Fix light timeouts and eclipse protection.

* Fix devp2p tests.

* Fix whisper-cli compilation.

* Fix compilation.

* Fix ethcore-sync tests.

* Revert "Fix light timeouts and eclipse protection."

This reverts commit 06285ea8c1d9d184d809f64b5507aece633da6cc.

* Increase timeouts.
2018-05-14 10:09:05 +02:00
Nicolas Gotchac
8b0ba97cf2 Refactoring ethcore-sync - Fixing warp-sync barrier (#8543)
* Start dividing sync chain : first supplier method

* WIP - updated chain sync supplier

* Finish refactoring the Chain Sync Supplier

* Create Chain Sync Requester

* Add Propagator for Chain Sync

* Add the Chain Sync Handler

* Move tests from mod -> handler

* Move tests to propagator

* Refactor SyncRequester arguments

* Refactoring peer fork header handler

* Fix wrong highest block number in snapshot sync

* Small refactor...

* Address PR grumbles

* Retry failed CI job

* Fix tests

* PR Grumbles
2018-05-09 12:05:34 +02:00