Commit Graph

46 Commits

Author SHA1 Message Date
rakita
aecc6fc862
Prometheus, heavy memory calls removed (#27) 2020-09-14 16:08:57 +02:00
rakita
506cee52e8
Fix issues during block sync (#11265) 2020-09-10 05:54:37 +03:00
Artem Vorotnikov
ea0c13c0a4
Remove private transactions 2020-09-10 05:54:37 +03:00
rakita
a8668b371c
Remove GetNodeData 2020-09-10 05:54:36 +03:00
Artem Vorotnikov
2ab8c72ce3
Correct internal par protocol nomenclature 2020-09-10 05:54:30 +03:00
adria0
725073a683
Fix warnings: docs 2020-08-07 20:47:13 +03:00
adria0
1700873f48
Fix warnings: dyn 2020-08-07 20:47:12 +03:00
Artem Vorotnikov
610d9baba4
Reformat the source code 2020-08-07 20:46:52 +03:00
s3krit
ff398fe7ff
V2.5.6 stable (#10961)
-  Fix cargo audit (#10921)
  - Add support for Energy Web Foundation's new chains (#10957)
  - Kaspersky AV whitelisting (#10919)
  - Avast whitelist script (#10900)
  - Docker images renaming (#10863)
  - Remove excessive warning (#10831)
  - Allow --nat extip:your.host.here.org (#10830)
  - When updating the client or when called from RPC, sleep should mean sleep (#10814)
  - added new ropsten-bootnode and removed old one (#10794)
  - ethkey no longer uses byteorder (#10786)
  - Do not drop the peer with None difficulty (#10772)
  - docs: Update Readme with TOC, Contributor Guideline. Update Cargo package descriptions (#10652)
2019-08-12 18:55:11 +02:00
elferdo
3adb640d2b
Bundle protocol and packet_id together in chain sync (#10315)
Define a new `enum` where devp2p subprotocol packet ids (currently eth and par) are defined. Additionally provide functionality to query id value and protocol of a given id object.
2019-02-13 09:20:33 +01:00
elferdo
046b8bbc8a
Make specification of protocol in SyncRequester::send_request explicit (#10295)
* Make the specification of the protocol to which a packet_id belongs to
  explicit when calling "SyncRequester::send_packet".

* Remove "SyncIO::send" and leave only "SyncIO::send_protocol"

* Adapt tests to new code.

* Strengthen tests to check if packet_id and protocol match when sending
a devp2p packet.
2019-02-08 13:01:29 +01:00
elferdo
b7e8621846 Increase number of requested block bodies in chain sync (#10247)
* Increase the number of block bodies requested during Sync.

* Increase the number of block bodies requested during Sync.

* Check if our peer is an older parity client with the bug
  of not handling large requests properly

* Add a ClientVersion struct and a ClientCapabilites trait

* Make ClientVersion its own module

* Refactor and extend use of ClientVersion

* Replace strings with ClientVersion in PeerInfo

* Group further functionality in ClientCapabilities

* Move parity client version data from tuple to its own struct.

* Implement accessor methods for ParityClientData and remove them
from ClientVersion.

* Minor fixes

* Make functions specific to parity return types specific to parity.

* Test for shorter ID strings

* Fix formatting and remove unneeded dependencies.

* Roll back Cargo.lock

* Commit last Cargo.lock

* Convert from string to ClientVersion

* * When checking if peer accepts service transactions just check
  if it's parity, remove version check.

* Remove dependency on semver in ethcore-sync

* Remove unnecessary String instantiation

* Rename peer_info to peer_version

* Update RPC test helpers

* Simplify From<String>

* Parse static version string only once

* Update RPC tests to new ClientVersion struct

* Document public members

* More robust parsing of ID string

* Minor changes.

* Update version in which large block bodies requests appear.

* Update ethcore/sync/src/block_sync.rs

Co-Authored-By: elferdo <elferdo@gmail.com>

* Update util/network/src/client_version.rs

Co-Authored-By: elferdo <elferdo@gmail.com>

* Update util/network/src/client_version.rs

Co-Authored-By: elferdo <elferdo@gmail.com>

* Update tests.

* Minor fixes.
2019-02-07 15:27:09 +01:00
Afri Schoedon
7c335e8764
misc: bump license header to 2019 (#10135)
* misc: bump license header to 2019

* misc: remove_duplicate_empty_lines.sh

* misc: run license header script

* commit cargo lock
2019-01-07 11:33:07 +01:00
Nicolas Gotchac
e435407080 Don't try to send oversized packets (#10042)
* Don't construct oversized packets

* Add test for payload limit

* [eth-sync] Fix wrongly computed data sizes

* Replace `MAX_RECEIPTS_TO_SEND` with overall softlimit
2019-01-04 19:58:21 +01:00
Anton Gavrilov
b180be7526 Private tx enabled flag added into STATUS packet (#9999)
* Add private tx enabled flag into status packet

* Error log added for the case with no peers available

* Add flag only for supported protocol versions

* Work with private handler refactored

* Log target changed

* Cargo.lock updated
2019-01-04 19:57:01 +01:00
Tomasz Drwięga
3650f2d51c Extract blockchain from ethcore (#10114)
* Split blockchain & db from ethcore.

* Clean up blockchain deps.

* Missing docs.

* Fix blockchain tests.

* Make other crates compile.

* Remove some re-exports.

* Remove types re-export from ethcore.

* Remove EVM dependency from transaction.

* Merge ethcore-transaction with common-types.

* Clean-up ethcore deps a bit.

* remove ethcore from cargo.toml

* Update ethcore/blockchain/src/lib.rs

Co-Authored-By: tomusdrw <tomusdrw@users.noreply.github.com>

* Address review comments.

* Update DB comment.

* Add tracking issue to the TODO and fix typo.

* Common naming for common types.

* Update ethcore/db/src/keys.rs

Co-Authored-By: tomusdrw <tomusdrw@users.noreply.github.com>

* Update ethcore/blockchain/src/generator.rs

Co-Authored-By: tomusdrw <tomusdrw@users.noreply.github.com>

* Try to fix beta tests.
2019-01-04 14:05:46 +01:00
joshua-mir
130901e820 Identity fix (#10128)
* fix #10125

fix service transaction version detection if --identity is enabled, change test to match how --identity actually works

* fix wrong var

* get the index of  v, not /

* idx, not idx.len()

* Update ethcore/sync/src/chain/propagator.rs

Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>

* Update ethcore/sync/src/chain/propagator.rs

Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>

* change version prefix to a const

* space

Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>
2019-01-04 13:23:46 +01:00
Nicolas Gotchac
5e9dc185a5 Fix unstable peers and slowness in sync (#9967)
* Don't sync all peers after each response

* Update formating

* Fix tests: add `continue_sync` to `Sync_step`

* Update ethcore/sync/src/chain/mod.rs

Co-Authored-By: ngotchac <ngotchac@gmail.com>
2018-11-28 12:19:35 +01:00
Tomasz Drwięga
0b5bbf6048 Improve block and transaction propagation (#9954)
* Refactor sync to add priority tasks.

* Send priority tasks notifications.

* Propagate blocks, optimize transactions.

* Implement transaction propagation. Use sync_channel.

* Tone down info.

* Prevent deadlock by not waiting forever for sync lock.

* Fix lock order.

* Don't use sync_channel to prevent deadlocks.

* Fix tests.
2018-11-28 11:30:05 +01:00
Nicolas Gotchac
9475a2e474 Keep existing blocks when restoring a Snapshot (#8643)
* Rename db_restore => client

* First step: make it compile!

* Second step: working implementation!

* Refactoring

* Fix tests

* PR Grumbles

* PR Grumbles WIP

* Migrate ancient blocks interating backward

* Early return in block migration if snapshot is aborted

* Remove RwLock getter (PR Grumble I)

* Remove dependency on `Client`: only used Traits

* Add test for recovering aborted snapshot recovery

* Add test for migrating old blocks

* Fix build

* PR Grumble I

* PR Grumble II

* PR Grumble III

* PR Grumble IV

* PR Grumble V

* PR Grumble VI

* Fix one test

* Fix test

* PR Grumble

* PR Grumbles

* PR Grumbles II

* Fix tests

* Release RwLock earlier

* Revert Cargo.lock

* Update _update ancient block_ logic: set local in `commit`

* Update typo in ethcore/src/snapshot/service.rs

Co-Authored-By: ngotchac <ngotchac@gmail.com>
2018-11-18 00:06:34 +01:00
David
c313039526 Update a few parity-common dependencies (#9663)
* Update a few parity-common dependencies

* cleanup

* cleanup

* revert update of ethereum/tests

* better reporting of network rlp errors

* Use rlp 0.3.0-beta.1

* fix util function get_dummy_blocks

* Already a Vec

* encode_list returns vec already

* Address grumble

* No need for betas

* Fix double spaces
2018-10-09 22:07:25 +02:00
Andrew Jones
4b6ebcbb61 Fix ancient blocks sync (#9531)
* Log block set in block_sync for easier debugging

* logging macros

* Match no args in sync logging macros

* Add QueueFull error

* Only allow importing headers if the first matches requested

* WIP

* Test for chain head gaps and log

* Calc distance even with 2 heads

* Revert previous commits, preparing simple fix

This reverts commit 5f38aa885b22ebb0e3a1d60120cea69f9f322628.

* Reject headers with no gaps when ChainHead

* Reset block sync download when queue full

* Simplify check for subchain heads

* Add comment to explain subchain heads filter

* Fix is_subchain_heads check and comment

* Prevent premature round completion after restart

This is a problem on mainnet where multiple stale peer requests will
force many rounds to complete quickly, forcing the retraction.

* Reset stale old blocks request after queue full

* Revert "Reject headers with no gaps when ChainHead"

This reverts commit 0eb865539e5dee37ab34f168f5fb643300de5ace.

* Add BlockSet to BlockDownloader logging

Currently it is difficult to debug this because there are two instances,
one for OldBlocks and one for NewBlocks. This adds the BlockSet to all
log messages for easy log filtering.

* Reset OldBlocks download from last enqueued

Previously when the ancient block queue was full it would restart the
download from the last imported block, so the ones still in the queue would be
redownloaded. Keeping the existing downloader instance and just
resetting it will start again from the last enqueued block.:wq

* Ignore expired Body and Receipt requests

* Log when ancient block download being restarted

* Only request old blocks from peers with >= difficulty

https://github.com/paritytech/parity-ethereum/pull/9226 might be too
permissive and causing the behaviour of the retraction soon after the
fork block. With this change the peer difficulty has to be greater than
or euqal to our syncing difficulty, so should still fix
https://github.com/paritytech/parity-ethereum/issues/9225

* Some logging and clear stalled blocks head

* Revert "Some logging and clear stalled blocks head"

This reverts commit 757641d9b817ae8b63fec684759b0815af9c4d0e.

* Reset stalled header if useless more than once

* Store useless headers in HashSet

* Add sync target to logging macro

* Don't disable useless peer and fix log macro

* Clear useless headers on reset and comments

* Use custom error for collecting blocks

Previously we resued BlockImportError, however only the Invalid case and
this made little sense with the QueueFull error.

* Remove blank line

* Test for reset sync after consecutive useless headers

* Don't reset after consecutive headers when chain head

* Delete commented out imports

* Return DownloadAction from collect_blocks instead of error

* Don't reset after round complete, was causing test hangs

* Add comment explaining reset after useless

* Replace HashSet with counter for useless headers

* Refactor sync reset on bad block/queue full

* Add missing target for log message

* Fix compiler errors and test after merge

* ethcore: revert ethereum tests submodule update
2018-10-09 15:31:40 +02:00
André Silva
dc14cce7a9 ethcore: fix detection of major import (#9552)
* sync: set state to idle after sync is completed

* sync: refactor sync reset
2018-10-08 12:04:38 +01:00
Jim Posen
2fc1679886 Verify block syncing responses against requests (#9670)
* sync: Validate received BlockHeaders packets against stored request.

* sync: Validate received BlockBodies and BlockReceipts.

* sync: Fix broken tests.

* sync: Unit tests for BlockDownloader::import_headers.

* sync: Unit tests for import_{bodies,receipts}.

* tests: Add missing method doc.
2018-10-03 18:35:10 +08:00
Tomasz Drwięga
cc963d42a0 RPC: parity_getBlockReceipts (#9527)
* Block receipts RPC.

* Use lazy evaluation of block receipts (ecrecover).

* Optimize transaction_receipt to prevent performance regression.

* Fix RPC grumbles.

* Add block & transaction receipt tests.

* Fix conversion to block id.
2018-09-25 18:06:14 +01:00
Marek Kotewicz
b57607e7d3
simplify ethcore errors by removing BlockImportError (#9593) 2018-09-24 11:28:54 +01:00
Nicolas Gotchac
753fd4bda3 Improve P2P discovery (#9526)
* Add `target` to Rust traces

* network-devp2p: Don't remove discovery peer in main sync

* network-p2p: Refresh discovery more often

* Update Peer discovery protocol

* Run discovery more often when not enough nodes connected

* Start the first discovery early

* Update fast discovery rate

* Fix tests

* Fix `ping` tests

* Fixing remote Node address ; adding PingPong round

* Fix tests: update new +1 PingPong round

* Increase slow Discovery rate
Check in flight FindNode before pings

* Add `deprecated` to deprecated_echo_hash

* Refactor `discovery_round` branching
2018-09-14 22:18:03 +02:00
Andrew Jones
bc53e78a04
Remove unused BlockStatus::Pending (#9447)
Pending case never instantiated, and only ever matched together with Unknown
2018-08-31 12:13:01 +01:00
Anton Gavrilov
1073d56245 Private packets verification and queue refactoring (#8715)
* Verify private transaction before propagating

* Private transactions queue reworked with tx pool queue direct usage

* Styling fixed

* Prevent resending private packets to the sender

* Process signed private transaction packets via io queue

* Test fixed

* Build and test fixed after merge

* Comments after review fixed

* Signed transaction taken from verified

* Fix after merge

* Pool scoring generalized in order to use externally

* Lib refactored according to the review comments

* Ready state refactored

* Redundant bound and copying removed

* Fixed build after the merge

* Forgotten case reworked

* Review comments fixed

* Logging reworked, target added

* Fix after merge
2018-08-29 20:31:04 +08:00
Marek Kotewicz
b87c7cac54 block view! removal in progress (#9397) 2018-08-24 10:53:31 +01:00
Tomasz Drwięga
29125e830b Lower the max size of transaction packet to prevent going oversize. (#9308)
* Lower the max size of transaction packet to prevent going oversize.

* Log RLP size.
2018-08-14 17:20:29 +02:00
David
e2095d4a5d Move ethereum-specific H256FastMap type to own crate (#9307)
* Add a `fastmap` crate that provides the H256FastMap specialized HashMap

* Use `fastmap` instead of `plain_hasher`

* Update submodules for Reasons™

* Submodule update
2018-08-09 09:51:48 +02:00
Tomasz Drwięga
90d7823acb Propagate transactions for next 4 blocks. (#9265)
Closes #9255 

This PR also removes the limit of max 64 transactions per packet, currently we only attempt to prevent the packet size to go over 8MB. This will only be the case for super-large transactions or high-block-gas-limit chains.

Patching this is important only for chains that have blocks that can fit more than 4k transactions (over 86M block gas limit)

For mainnet, we should actually see a tiny bit faster propagation since instead of computing 4k pending set, we only need `4 * 8M / 21k = 1523` transactions.

Running some tests on `dekompile` node right now, to check how it performs in the wild.
2018-08-02 11:58:02 +01:00
Marek Kotewicz
b4ae1b6528 decode block rlp less often (#9252)
in total:
- removed 4 redundant rlp deserializations
- avoid 1 redundant block data copy
2018-08-02 10:20:46 +01:00
Andrew Jones
fb503f523b
Allow old blocks from peers with lower difficulty (#9226)
Previously we only allow downloading of old blocks if the peer
difficulty was greater than our syncing difficulty. This change allows
downloading of blocks from peers where the difficulty is greater then
the last downloaded old block.
2018-07-27 10:13:05 +01:00
Marek Kotewicz
073365d5d9
handle SyncHandler errors properly (#9151)
* handle SyncHandler errors properly, closes #9150

* applied review suggestions
2018-07-19 12:46:33 +02:00
Afri Schoedon
484ecfaf47
Parity Ethereum 2.0.0 (#9052)
* parity-version: major bump to 2.0.0 🎉

* parity-ethereum: rename crate 🌵

* ethcore: only accept service transactions from parity-ethereum nodes

* parity: fix --identity tests

* rpc: fix sync provider in tests

* rpc: fix parity_net_peers test

* ethcore-sync: accept service transactions from parity and parity-ethereum

* ethcore-sync: fix indentation

* ethcore-sync: split the ifs to reduce code redundancy

* ethcore-sync: fix syntax

* Fix building ethcore

* update cargo.lock

* parity-version: major bump to 2.0.0 tada

* fix merge
2018-07-11 13:35:10 +02:00
Tomasz Drwięga
78e001284f Recently rejected cache for transaction queue (#9005)
* Store recently rejected transactions.

* Don't cache AlreadyImported rejections.

* Make the size of transaction verification queue dependent on pool size.

* Add a test for recently rejected.

* Fix logging for recently rejected.

* Make rejection cache smaller.

* obsolete test removed

* obsolete test removed

* Construct cache with_capacity.
2018-07-02 18:00:06 +01:00
David
9caa868603
Make HashDB generic (#8739)
The `patricia_trie` crate is generic over the hasher (by way of HashDB) and node encoding scheme. Adds a new `patricia_trie_ethereum` crate with concrete impls for Keccak/RLP.
2018-07-02 18:50:05 +02:00
André Silva
3016d54f13 Minor fix in chain supplier and light provider (#8906)
* fix chain supplier increment

* fix light provider block_headers
2018-06-15 17:30:34 +02:00
Tomasz Drwięga
4938d5dde5 Limit the number of transactions in pending set (#8777)
* Unordered iterator.

* Use unordered and limited set if full not required.

* Split timeout work into smaller timers.

* Avoid collecting all pending transactions when mining

* Remove println.

* Use priority ordering in eth-filter.

* Fix ethcore-miner tests and tx propagation.

* Review grumbles addressed.

* Add test for unordered not populating the cache.

* Fix ethcore tests.

* Fix light tests.

* Fix ethcore-sync tests.

* Fix RPC tests.
2018-06-12 08:22:54 +02:00
Niklas Adolfsson
98b7c07171 Update license header and scripts (#8666)
* Update `add_license` script

* run script

* add `remove duplicate lines script` and run it

* Revert changes `English spaces`

* strip whitespaces

* Revert `GPL` in files with `apache/mit license`

* don't append `gpl license` in files with other lic

* Don't append `gpl header` in files with other lic.

* re-ran script

* include c and cpp files too

* remove duplicate header

* rebase nit
2018-06-04 10:19:50 +02:00
André Silva
2060ea5de3 ethcore-sync: fix connection to peers behind chain fork block (#8710) 2018-06-02 10:29:44 +02:00
Nicolas Gotchac
cdbcfaa7de Resumable warp-sync / Seed downloaded snapshots (#8544)
* Start dividing sync chain : first supplier method

* WIP - updated chain sync supplier

* Finish refactoring the Chain Sync Supplier

* Create Chain Sync Requester

* Add Propagator for Chain Sync

* Add the Chain Sync Handler

* Move tests from mod -> handler

* Move tests to propagator

* Refactor SyncRequester arguments

* Refactoring peer fork header handler

* Fix wrong highest block number in snapshot sync

* Small refactor...

* Resume warp-sync downloaded chunks

* Add comments

* Refactoring the previous chunks import

* Fix tests

* Address PR grumbles

* Fix not seeding current snapshot

* Address PR Grumbles

* Address PR grumble

* Retry failed CI job

* Update SnapshotService readiness check
Fix restoration locking issue for previous chunks restoration

* Fix tests

* Fix tests

* Fix test

* Early abort importing previous chunks

* PR Grumbles

* Update Gitlab CI config

* SyncState back to Waiting when Manifest peers disconnect

* Move fix

* Better fix

* Revert GitLab CI changes

* Fix Warning

* Refactor resuming snapshots

* Fix string construction

* Revert "Refactor resuming snapshots"

This reverts commit 75fd4b553a38e4a49dc5d6a878c70e830ff382eb.

* Update informant log

* Fix string construction

* Refactor resuming snapshots

* Fix informant

* PR Grumbles

* Update informant message : show chunks done

* PR Grumbles

* Fix

* Fix Warning

* PR Grumbles
2018-05-16 22:01:55 +02:00
Tomasz Drwięga
08abf67a51 Fix packet count when talking with PAR2 peers (#8555)
* Support diferent packet counts in different protocol versions.

* Fix light timeouts and eclipse protection.

* Fix devp2p tests.

* Fix whisper-cli compilation.

* Fix compilation.

* Fix ethcore-sync tests.

* Revert "Fix light timeouts and eclipse protection."

This reverts commit 06285ea8c1d9d184d809f64b5507aece633da6cc.

* Increase timeouts.
2018-05-14 10:09:05 +02:00
Nicolas Gotchac
8b0ba97cf2 Refactoring ethcore-sync - Fixing warp-sync barrier (#8543)
* Start dividing sync chain : first supplier method

* WIP - updated chain sync supplier

* Finish refactoring the Chain Sync Supplier

* Create Chain Sync Requester

* Add Propagator for Chain Sync

* Add the Chain Sync Handler

* Move tests from mod -> handler

* Move tests to propagator

* Refactor SyncRequester arguments

* Refactoring peer fork header handler

* Fix wrong highest block number in snapshot sync

* Small refactor...

* Address PR grumbles

* Retry failed CI job

* Fix tests

* PR Grumbles
2018-05-09 12:05:34 +02:00