* Move snapshot to own crate
Sort out imports
* WIP cargo toml
* Make snapshotting generic over the client
Sort out tests
* Sort out types from blockchain and client
* Sort out sync
* Sort out imports and generics
* Sort out main binary
* Fix sync test-helpers
* Sort out import for secret-store
* Sort out more imports
* Fix easy todos
* cleanup
* Cleanup
* remove unneded workspace member
* cleanup
* Sort out test-helpers dependency on account-db
* Update ethcore/client-traits/src/lib.rs
Co-Authored-By: Niklas Adolfsson <niklasadolfsson1@gmail.com>
* Update ethcore/snapshot/Cargo.toml
* Add client-traits crate
Move the BlockInfo trait to new crate
* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.
* Use new machine and client-traits crates in ethcore
* Use new crates machine and client-traits instead of ethcore where appropriate
* Fix tests
* Don't re-export so many types from ethcore::client
* Fixing more fallout from removing re-export
* fix test
* More fallout from not re-exporting types
* Add some docs
* cleanup
* import the macro edition style
* Tweak docs
* Add missing import
* remove unused ethabi_derive imports
* Use latest ethabi-contract
* Move many traits from ethcore/client/traits to client-traits crate
Initial version of extracted Engine trait
* Move snapshot related traits to the engine crate (eew)
* Move a few snapshot related types to common_types
Cleanup Executed as exported from machine crate
* fix warning
* Gradually introduce new engine crate: snapshot
* ethcore typechecks with new engine crate
* Sort out types outside ethcore
* Add an EpochVerifier to ethash and use that in Engine.epoch_verifier()
Cleanup
* Document pub members
* Sort out tests
Sort out default impls for EpochVerifier
* Add test-helpers feature and move EngineSigner impl to the right place
* Sort out tests
* Sort out tests and refactor verification types
* Fix missing traits
* More missing traits
Fix Histogram
* Fix tests and cleanup
* cleanup
* Put back needed logger import
* Don't rexport common_types from ethcore/src/client
Don't export ethcore::client::*
* Remove files no longer used
Use types from the engine crate
Explicit exports from engine::engine
* Get rid of itertools
* Move a few more traits from ethcore to client-traits: BlockChainReset, ScheduleInfo, StateClient
* Move ProvingBlockChainClient to client-traits
* Don't re-export ForkChoice and Transition from ethcore
* Address grumbles: sort imports, remove commented out code
* Fix merge resolution error
* Extract the Clique engine to own crate
* Extract NullEngine and the block_reward module from ethcore
* Extract InstantSeal engine to own crate
* Extract remaining engines
* Extract executive_state to own crate so it can be used by engine crates
* Remove snapshot stuff from the engine crate
* Put snapshot traits back in ethcore
* cleanup
* Remove stuff from ethcore
* Don't use itertools
* itertools in aura is legit-ish
* More post-merge fixes
* Re-export less types in client
* cleanup
* Update ethcore/block-reward/Cargo.toml
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update ethcore/engines/basic-authority/Cargo.toml
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update ethcore/engines/ethash/Cargo.toml
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update ethcore/engines/clique/src/lib.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* signers is already a ref
* Add an EngineType enum to tighten up Engine.name()
* Introduce Snapshotting enum to distinguish the type of snapshots a chain uses
* Rename supports_warp to snapshot_mode
* Missing import
* Update ethcore/src/snapshot/consensus/mod.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* remove double-semicolons
* Temp storage for the private state added
* Temp storage for the private state added
* Request message added
* Store and retrieve offchain state logic
* State sync cache
* Private state column added to key value db
* Private state column added to key value db
* Indexing stored states via its hash
* Works with errors changed
* Private state stored into the local db
* Access to private state db added to sync io
* Private state db file added
* Rlp packets for retrieiving private state data added
* Handling of private sync completed message
* Test code fixed
* External flag for offchain storing added
* Test for private state sync added
* Saving private state logic corrected
* Migration code corrected
* Fixes after merge with master
* Merge with head
* Additional checks for slices
* Log for private state retrieval added
* Limit time of retrieving private states
* Store required hashes for every request, mark them stale if needed
* Store requested private state hashes and check received data
* Log stale requests
* State insertion fix
* Refactoring of how logging passed to state store
* Heapsize removed, syncing hashes structure reworked
* Check state length returned by contract
* Get rid of OverlayDB
* hash-db version updated
* Test fixed
* One more test fixed
* Add client-traits crate
Move the BlockInfo trait to new crate
* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.
* Use new machine and client-traits crates in ethcore
* Use new crates machine and client-traits instead of ethcore where appropriate
* Fix tests
* Don't re-export so many types from ethcore::client
* Fixing more fallout from removing re-export
* fix test
* More fallout from not re-exporting types
* Add some docs
* cleanup
* import the macro edition style
* Tweak docs
* Add missing import
* remove unused ethabi_derive imports
* Use latest ethabi-contract
* Move many traits from ethcore/client/traits to client-traits crate
Initial version of extracted Engine trait
* Move snapshot related traits to the engine crate (eew)
* Move a few snapshot related types to common_types
Cleanup Executed as exported from machine crate
* fix warning
* Gradually introduce new engine crate: snapshot
* ethcore typechecks with new engine crate
* Sort out types outside ethcore
* Add an EpochVerifier to ethash and use that in Engine.epoch_verifier()
Cleanup
* Document pub members
* Sort out tests
Sort out default impls for EpochVerifier
* Add test-helpers feature and move EngineSigner impl to the right place
* Sort out tests
* Sort out tests and refactor verification types
* Fix missing traits
* More missing traits
Fix Histogram
* Fix tests and cleanup
* cleanup
* Put back needed logger import
* Don't rexport common_types from ethcore/src/client
Don't export ethcore::client::*
* Remove files no longer used
Use types from the engine crate
Explicit exports from engine::engine
* Get rid of itertools
* Move a few more traits from ethcore to client-traits: BlockChainReset, ScheduleInfo, StateClient
* Move ProvingBlockChainClient to client-traits
* Don't re-export ForkChoice and Transition from ethcore
* Address grumbles: sort imports, remove commented out code
* Fix merge resolution error
* merge failure
* Add client-traits crate
Move the BlockInfo trait to new crate
* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.
* Use new machine and client-traits crates in ethcore
* Use new crates machine and client-traits instead of ethcore where appropriate
* Fix tests
* Don't re-export so many types from ethcore::client
* Fixing more fallout from removing re-export
* fix test
* More fallout from not re-exporting types
* Add some docs
* cleanup
* import the macro edition style
* Tweak docs
* Add missing import
* remove unused ethabi_derive imports
* Use latest ethabi-contract
* WIP move errors, pod_account and state account to own crates
* Sort out dependencies, fix broken code and tests
Remove botched ethcore-error crate
* remove template line
* fix review feedback
* Remove test-only AccountDBMut::new
* Extract AccountDB to account-db
* Move Substate to state-account – wip
* Add lib.rs
* cleanup
* test failure
* test failure 2
* third time's the charm
* Add factories crate
* Use new factories crate
* Use factories crate
* Extract trace
* Fix tests
* Sort out parity-util-mem and parking_lot
* cleanup
* WIP port over the rest of state from ethcore
* Collect all impls for Machine
* some notes
* Rename pod-account to pod
* Move PodState to pod crate
* Use PodState from pod crate
* Fix use clause for json tests
* Sort out evmbin
* Add missing code and use PodState
* Move code that depends on Machine and Executive to own module
* Sort out cloning errors, fix ethcore to use new state crate
* Do without funky From impls
* Fix ethcore tests
* Fixes around the project to use new state crate
* Add back the more specific impls of StateOrBlock From conversions
* Move execute to freestanding function and remove it from trait
Sort out the error handling in executive_state by moving the result types from state to ethcore
Undo the verbose code added to work around the StateOrBlock From conversions
* cleanup
* Fix "error: enum variants on type aliases are experimental"
* Bring back the state tests
Fix whitespace
* remove ethcore/state/mod.rs
* cleanup
* cleanup
* Cleanup state-account errors
* Fix more todos
Add module docs
* Add error.rs
* Fixup Cargo.lock
* Smaller ethcore API is fine
* Add `to-pod-full` feature to state-account
Fix evmbin
* Fix a few more test failures
* Fix RPC test build
* Baptize the new trait
* Remove resolved TODOs
* Rename state-account to account-state
* Do not re-export the trace crate
* Don't export state_db from ethcore
* Let private-tx use StateDB. :(
* Remove ethcore/src/pod_state.rs
* Inner type does not need to be pub/pub(crate)
* optimise imports
* Revert "Inner type does not need to be pub/pub(crate)"
This reverts commit 2f839f8a0f72f71334da64620f57e6dd6039f06b.
* Move DatabaseExtras to ethcore-blockchain
* Add database_extra module to ethcore-blockchain
* Remove to-pod-full feature
* cosmetics
* New crate: state-db
* Add new crate
* Move PreverifiedBlock and BlockError to types
* Sort out the merge
* Add missing `license` meta data keys
* wip
* wip client-traits
* merge conflict
* verification crate type checks
* Move impls for CommonParams to common_types
Fix misc stuff in ethcore
* Fix tests
* Implement VerifyingEngine for all engines except Ethash
Temporarily sort out error handling
Move more types to common_types
* Split Engine in two and move code around
* cleanup
* verification: don't rexport common_types
* Use error from common_types
* Consolidate error types
* VerifyingEngine use Errors from common_types
* verification: Use error type from common_types
* SnapshotError moved to common_types
* Move more code from Engne to VerifyingEngine
Add a VerifyingClient trait: BlockInfo + CallContract
Whitespace
* Add MAX_UNCLE_AGE const
* Port over remaining code from ethcore/verification
* Use errors from common_types
* Fix the confusing "io" naming
* Move more types into common_types
* Add todos
* Experiment with Engine trait outside ethcore
* Hook up types from common_types in ethcore
Don't use verification crate
Don't use client-traits crate
* Revert to impl Engine for Arc<Ethash> and add note to explain why
Revert moving ClientIoMessage to common_types
Fix build
* Remove ClientIoMessage from common_types
* Cleanup
* More cleanup
* Sort error handling changes in the rest of parity
* Remove unused code
* Remove WIP types
* Cleanup todos not tackled here
* remove cruft
* Fix some whitespace and a merge error
* ethcore tests
* test failures
* Restore Engine impls to master to make review a bit easier
* cleanup
* whitespace
* applied review suggestions
* types does not depend on rustc-hex
* ethash engine moved to engine module
* applied review suggestion
* Run cargo fix on `vm`
* Run cargo fix on ethcore-db
* Run cargo fix on evm
* Run cargo fix on ethcore-light
* Run cargo fix on journaldb
* Run cargo fix on wasm
* Missing docs
* Run cargo fix on ethcore-sync
* WIP move errors, pod_account and state account to own crates
* Sort out dependencies, fix broken code and tests
Remove botched ethcore-error crate
* remove template line
* fix review feedback
* Remove test-only AccountDBMut::new
* Extract AccountDB to account-db
* Move Substate to state-account – wip
* Add lib.rs
* cleanup
* test failure
* test failure 2
* third time's the charm
* Add factories crate
* Use new factories crate
* Use factories crate
* Extract trace
* Fix tests
* Sort out parity-util-mem and parking_lot
* cleanup
* WIP port over the rest of state from ethcore
* Collect all impls for Machine
* some notes
* Rename pod-account to pod
* Move PodState to pod crate
* Use PodState from pod crate
* Fix use clause for json tests
* Sort out evmbin
* Add missing code and use PodState
* Move code that depends on Machine and Executive to own module
* Sort out cloning errors, fix ethcore to use new state crate
* Do without funky From impls
* Fix ethcore tests
* Fixes around the project to use new state crate
* Add back the more specific impls of StateOrBlock From conversions
* Move execute to freestanding function and remove it from trait
Sort out the error handling in executive_state by moving the result types from state to ethcore
Undo the verbose code added to work around the StateOrBlock From conversions
* cleanup
* Fix "error: enum variants on type aliases are experimental"
* Bring back the state tests
Fix whitespace
* remove ethcore/state/mod.rs
* cleanup
* cleanup
* Cleanup state-account errors
* Fix more todos
Add module docs
* Add error.rs
* Fixup Cargo.lock
* Smaller ethcore API is fine
* Add `to-pod-full` feature to state-account
Fix evmbin
* Fix a few more test failures
* Fix RPC test build
* Baptize the new trait
* Remove resolved TODOs
* Rename state-account to account-state
* Do not re-export the trace crate
* Don't export state_db from ethcore
* Let private-tx use StateDB. :(
* Remove ethcore/src/pod_state.rs
* Inner type does not need to be pub/pub(crate)
* optimise imports
* Revert "Inner type does not need to be pub/pub(crate)"
This reverts commit 2f839f8a0f72f71334da64620f57e6dd6039f06b.
* Move DatabaseExtras to ethcore-blockchain
* Add database_extra module to ethcore-blockchain
* Remove to-pod-full feature
* Sort out the merge
* sort imports
* address grumbles
* rename crate
* address more grumbles
* Better logging when backfilling ancient blocks fail
Print total blocks imported, closes#10792
* `finalize()` doesn't need Engine
Pull out call to migrated_blocks() from replace_client_db()
* More logs
* Clarify that the percentage may be misleading
* Remove replace_client_db() and replace with a straight call to restore_db()
* Include the parent_hash in UnlinkedAncientBlockChain errors
* Add a new RestorationStatus varian: Finalizing (as it can take a loooong while)
Call abort_restore() when restoration fails
* Add missing cases for new variant
* typos
* Typo and derive Debug
* Do not attempt to salvage existing blocks unless they form a complete chain back to genesis
* Fix test
* Revert "Fix test"
This reverts commit f027d4b4cb7b6c23fceec528c1711886ba9cfe4e.
* Fix test again
* Update comment
* Be careful about locks
* fix test failure
* Do not defer returning an error when the chain is broken
* Review feedback
* no hex formatting for Option
Define a new `enum` where devp2p subprotocol packet ids (currently eth and par) are defined. Additionally provide functionality to query id value and protocol of a given id object.
* Make the specification of the protocol to which a packet_id belongs to
explicit when calling "SyncRequester::send_packet".
* Remove "SyncIO::send" and leave only "SyncIO::send_protocol"
* Adapt tests to new code.
* Strengthen tests to check if packet_id and protocol match when sending
a devp2p packet.
* Increase the number of block bodies requested during Sync.
* Increase the number of block bodies requested during Sync.
* Check if our peer is an older parity client with the bug
of not handling large requests properly
* Add a ClientVersion struct and a ClientCapabilites trait
* Make ClientVersion its own module
* Refactor and extend use of ClientVersion
* Replace strings with ClientVersion in PeerInfo
* Group further functionality in ClientCapabilities
* Move parity client version data from tuple to its own struct.
* Implement accessor methods for ParityClientData and remove them
from ClientVersion.
* Minor fixes
* Make functions specific to parity return types specific to parity.
* Test for shorter ID strings
* Fix formatting and remove unneeded dependencies.
* Roll back Cargo.lock
* Commit last Cargo.lock
* Convert from string to ClientVersion
* * When checking if peer accepts service transactions just check
if it's parity, remove version check.
* Remove dependency on semver in ethcore-sync
* Remove unnecessary String instantiation
* Rename peer_info to peer_version
* Update RPC test helpers
* Simplify From<String>
* Parse static version string only once
* Update RPC tests to new ClientVersion struct
* Document public members
* More robust parsing of ID string
* Minor changes.
* Update version in which large block bodies requests appear.
* Update ethcore/sync/src/block_sync.rs
Co-Authored-By: elferdo <elferdo@gmail.com>
* Update util/network/src/client_version.rs
Co-Authored-By: elferdo <elferdo@gmail.com>
* Update util/network/src/client_version.rs
Co-Authored-By: elferdo <elferdo@gmail.com>
* Update tests.
* Minor fixes.
* Add private tx enabled flag into status packet
* Error log added for the case with no peers available
* Add flag only for supported protocol versions
* Work with private handler refactored
* Log target changed
* Cargo.lock updated
* fix#10125
fix service transaction version detection if --identity is enabled, change test to match how --identity actually works
* fix wrong var
* get the index of v, not /
* idx, not idx.len()
* Update ethcore/sync/src/chain/propagator.rs
Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>
* Update ethcore/sync/src/chain/propagator.rs
Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>
* change version prefix to a const
* space
Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>
* Rename db_restore => client
* First step: make it compile!
* Second step: working implementation!
* Refactoring
* Fix tests
* PR Grumbles
* PR Grumbles WIP
* Migrate ancient blocks interating backward
* Early return in block migration if snapshot is aborted
* Remove RwLock getter (PR Grumble I)
* Remove dependency on `Client`: only used Traits
* Add test for recovering aborted snapshot recovery
* Add test for migrating old blocks
* Fix build
* PR Grumble I
* PR Grumble II
* PR Grumble III
* PR Grumble IV
* PR Grumble V
* PR Grumble VI
* Fix one test
* Fix test
* PR Grumble
* PR Grumbles
* PR Grumbles II
* Fix tests
* Release RwLock earlier
* Revert Cargo.lock
* Update _update ancient block_ logic: set local in `commit`
* Update typo in ethcore/src/snapshot/service.rs
Co-Authored-By: ngotchac <ngotchac@gmail.com>
* Log block set in block_sync for easier debugging
* logging macros
* Match no args in sync logging macros
* Add QueueFull error
* Only allow importing headers if the first matches requested
* WIP
* Test for chain head gaps and log
* Calc distance even with 2 heads
* Revert previous commits, preparing simple fix
This reverts commit 5f38aa885b22ebb0e3a1d60120cea69f9f322628.
* Reject headers with no gaps when ChainHead
* Reset block sync download when queue full
* Simplify check for subchain heads
* Add comment to explain subchain heads filter
* Fix is_subchain_heads check and comment
* Prevent premature round completion after restart
This is a problem on mainnet where multiple stale peer requests will
force many rounds to complete quickly, forcing the retraction.
* Reset stale old blocks request after queue full
* Revert "Reject headers with no gaps when ChainHead"
This reverts commit 0eb865539e5dee37ab34f168f5fb643300de5ace.
* Add BlockSet to BlockDownloader logging
Currently it is difficult to debug this because there are two instances,
one for OldBlocks and one for NewBlocks. This adds the BlockSet to all
log messages for easy log filtering.
* Reset OldBlocks download from last enqueued
Previously when the ancient block queue was full it would restart the
download from the last imported block, so the ones still in the queue would be
redownloaded. Keeping the existing downloader instance and just
resetting it will start again from the last enqueued block.:wq
* Ignore expired Body and Receipt requests
* Log when ancient block download being restarted
* Only request old blocks from peers with >= difficulty
https://github.com/paritytech/parity-ethereum/pull/9226 might be too
permissive and causing the behaviour of the retraction soon after the
fork block. With this change the peer difficulty has to be greater than
or euqal to our syncing difficulty, so should still fix
https://github.com/paritytech/parity-ethereum/issues/9225
* Some logging and clear stalled blocks head
* Revert "Some logging and clear stalled blocks head"
This reverts commit 757641d9b817ae8b63fec684759b0815af9c4d0e.
* Reset stalled header if useless more than once
* Store useless headers in HashSet
* Add sync target to logging macro
* Don't disable useless peer and fix log macro
* Clear useless headers on reset and comments
* Use custom error for collecting blocks
Previously we resued BlockImportError, however only the Invalid case and
this made little sense with the QueueFull error.
* Remove blank line
* Test for reset sync after consecutive useless headers
* Don't reset after consecutive headers when chain head
* Delete commented out imports
* Return DownloadAction from collect_blocks instead of error
* Don't reset after round complete, was causing test hangs
* Add comment explaining reset after useless
* Replace HashSet with counter for useless headers
* Refactor sync reset on bad block/queue full
* Add missing target for log message
* Fix compiler errors and test after merge
* ethcore: revert ethereum tests submodule update
* sync: Validate received BlockHeaders packets against stored request.
* sync: Validate received BlockBodies and BlockReceipts.
* sync: Fix broken tests.
* sync: Unit tests for BlockDownloader::import_headers.
* sync: Unit tests for import_{bodies,receipts}.
* tests: Add missing method doc.
* Verify private transaction before propagating
* Private transactions queue reworked with tx pool queue direct usage
* Styling fixed
* Prevent resending private packets to the sender
* Process signed private transaction packets via io queue
* Test fixed
* Build and test fixed after merge
* Comments after review fixed
* Signed transaction taken from verified
* Fix after merge
* Pool scoring generalized in order to use externally
* Lib refactored according to the review comments
* Ready state refactored
* Redundant bound and copying removed
* Fixed build after the merge
* Forgotten case reworked
* Review comments fixed
* Logging reworked, target added
* Fix after merge
* Add a `fastmap` crate that provides the H256FastMap specialized HashMap
* Use `fastmap` instead of `plain_hasher`
* Update submodules for Reasons™
* Submodule update
Closes#9255
This PR also removes the limit of max 64 transactions per packet, currently we only attempt to prevent the packet size to go over 8MB. This will only be the case for super-large transactions or high-block-gas-limit chains.
Patching this is important only for chains that have blocks that can fit more than 4k transactions (over 86M block gas limit)
For mainnet, we should actually see a tiny bit faster propagation since instead of computing 4k pending set, we only need `4 * 8M / 21k = 1523` transactions.
Running some tests on `dekompile` node right now, to check how it performs in the wild.
Previously we only allow downloading of old blocks if the peer
difficulty was greater than our syncing difficulty. This change allows
downloading of blocks from peers where the difficulty is greater then
the last downloaded old block.
* Store recently rejected transactions.
* Don't cache AlreadyImported rejections.
* Make the size of transaction verification queue dependent on pool size.
* Add a test for recently rejected.
* Fix logging for recently rejected.
* Make rejection cache smaller.
* obsolete test removed
* obsolete test removed
* Construct cache with_capacity.
The `patricia_trie` crate is generic over the hasher (by way of HashDB) and node encoding scheme. Adds a new `patricia_trie_ethereum` crate with concrete impls for Keccak/RLP.
* Unordered iterator.
* Use unordered and limited set if full not required.
* Split timeout work into smaller timers.
* Avoid collecting all pending transactions when mining
* Remove println.
* Use priority ordering in eth-filter.
* Fix ethcore-miner tests and tx propagation.
* Review grumbles addressed.
* Add test for unordered not populating the cache.
* Fix ethcore tests.
* Fix light tests.
* Fix ethcore-sync tests.
* Fix RPC tests.
* Update `add_license` script
* run script
* add `remove duplicate lines script` and run it
* Revert changes `English spaces`
* strip whitespaces
* Revert `GPL` in files with `apache/mit license`
* don't append `gpl license` in files with other lic
* Don't append `gpl header` in files with other lic.
* re-ran script
* include c and cpp files too
* remove duplicate header
* rebase nit
