* Only use kvdb "column families"
This PR contains the changes necessary to use the `kvdb-*` crates from https://github.com/paritytech/parity-common/pull/278 (so a synchronized merge is required) which drops support for the old-style rocksdb "default" column to get a smaller and less complex API.
As it stands this PR is working correctly except for secret-store; we need to migrate it to use a new column family.
* Fix secretstore build
* Fix secretstore build: include ethkey when building with the "accounts" feature
* typos
* Restore state test commit
* Override all of parity-common from git
* Be precise about version requirement to migrate secretstore code
* Update ethcore/db/src/db.rs
Co-Authored-By: Niklas Adolfsson <niklasadolfsson1@gmail.com>
* Address review grumbles
* Review grumbles
* Cleanup
Co-authored-by: Niklas Adolfsson <niklasadolfsson1@gmail.com>
* Comments and todos
Use `snapshot_sync` as logging target
* fix compilation
* More todos, more logs
* Fix picking snapshot peer: prefer the one with the highest block number
More docs, comments, todos
* Adjust WAIT_PEERS_TIMEOUT to be a multiple of MAINTAIN_SYNC_TIMER to try to fix snapshot startup problems
Docs, todos, comments
* Tabs
* Formatting
* Don't build new rlp::EMPTY_LIST_RLP instances
* Dial down debug logging
* Don't warn about missing hashes in the manifest: it's normal
Log client version on peer connect
* Cleanup
* Do not skip snapshots further away than 30k block from the highest block seen
Currently we look for peers that seed snapshots that are close to the highest block seen on the network (where "close" means withing 30k blocks). When a node starts up we wait for some time (5sec, increased here to 10sec) to let peers connect and if we have found a suitable peer to sync a snapshot from at the end of that delay, we start the download; if none is found and --warp-barrier is used we stall, otherwise we start a slow-sync.
When looking for a suitable snapshot, we use the highest block seen on the network to check if a peer has a snapshot that is within 30k blocks of that highest block number. This means that in a situation where all available snapshots are older than that, we will often fail to start a snapshot at all. What's worse is that the longer we delay starting a snapshot sync (to let more peers connect, in the hope of finding a good snapshot), the more likely we are to have seen a high block and thus the more likely we become to accept a snapshot.
This commit removes this comparison with the highest blocknumber criteria entirely and picks the best snapshot we find in 10sec.
* lockfile
* Add a `ChunkType::Dupe` variant so that we do not disconnect a peer if they happen to send us a duplicate chunk (just ignore the chunk and keep going)
Resolve some documentation todos, add more
* tweak log message
* Don't warp sync twice
Check if our own block is beyond the given warp barrier (can happen after we've completed a warp sync but are not quite yet synced up to the tip) and if so, don't sync.
More docs, resolve todos.
Dial down some `sync` debug level logging to trace
* Avoid iterating over all snapshot block/state hashes to find the next work item
Use a HashSet instead of a Vec and remove items from the set as chunks are processed. Calculate and store the total number of chunks in the `Snapshot` struct instead of counting pending chunks each time.
* Address review grumbles
* Log correct number of bytes written to disk
* Revert ChunkType::Dup change
* whitespace grumble
* Cleanup debugging code
* Fix docs
* Fix import and a typo
* Fix test impl
* Use `indexmap::IndexSet` to ensure chunk hashes are accessed in order
* Revert increased SNAPSHOT_MANIFEST_TIMEOUT: 5sec should be enough
* Update a few dependencies
Updates two dependencies: `kvdb-rocksdb` and `h2`. Brings in `parking_lot 0.9` which is unintended but possibly fine.
* Bump parking_lot to 0.9
Bump kvdb-memorydb to 0.2 (from git atm)
* New kvdb-memorydb is not breaking
* Remove [patch]
* [ethcore]: reduce re-exports
Last piece of refactoring to close#10130 after work done by @dvdplm et. al
After this PR, we have the following re-exports from other crates in `non-testbuilds`:
- evm::VMType (client)
- ethcore_miner::local_accounts::LocalAccounts (miner)
- ethcore_miner::pool::PendingOrdering (miner)
I think the miner re-exports make sense (closely tied to the module) and `VMType` (related to module but not closely)
* fix(grumbles): remove re-export `VerifierType`
* Move snapshot to own crate
Sort out imports
* WIP cargo toml
* Make snapshotting generic over the client
Sort out tests
* Sort out types from blockchain and client
* Sort out sync
* Sort out imports and generics
* Sort out main binary
* Fix sync test-helpers
* Sort out import for secret-store
* Sort out more imports
* Fix easy todos
* cleanup
* Cleanup
* remove unneded workspace member
* cleanup
* Sort out test-helpers dependency on account-db
* Update ethcore/client-traits/src/lib.rs
Co-Authored-By: Niklas Adolfsson <niklasadolfsson1@gmail.com>
* Update ethcore/snapshot/Cargo.toml
* Add client-traits crate
Move the BlockInfo trait to new crate
* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.
* Use new machine and client-traits crates in ethcore
* Use new crates machine and client-traits instead of ethcore where appropriate
* Fix tests
* Don't re-export so many types from ethcore::client
* Fixing more fallout from removing re-export
* fix test
* More fallout from not re-exporting types
* Add some docs
* cleanup
* import the macro edition style
* Tweak docs
* Add missing import
* remove unused ethabi_derive imports
* Use latest ethabi-contract
* Move many traits from ethcore/client/traits to client-traits crate
Initial version of extracted Engine trait
* Move snapshot related traits to the engine crate (eew)
* Move a few snapshot related types to common_types
Cleanup Executed as exported from machine crate
* fix warning
* Gradually introduce new engine crate: snapshot
* ethcore typechecks with new engine crate
* Sort out types outside ethcore
* Add an EpochVerifier to ethash and use that in Engine.epoch_verifier()
Cleanup
* Document pub members
* Sort out tests
Sort out default impls for EpochVerifier
* Add test-helpers feature and move EngineSigner impl to the right place
* Sort out tests
* Sort out tests and refactor verification types
* Fix missing traits
* More missing traits
Fix Histogram
* Fix tests and cleanup
* cleanup
* Put back needed logger import
* Don't rexport common_types from ethcore/src/client
Don't export ethcore::client::*
* Remove files no longer used
Use types from the engine crate
Explicit exports from engine::engine
* Get rid of itertools
* Move a few more traits from ethcore to client-traits: BlockChainReset, ScheduleInfo, StateClient
* Move ProvingBlockChainClient to client-traits
* Don't re-export ForkChoice and Transition from ethcore
* Address grumbles: sort imports, remove commented out code
* Fix merge resolution error
* Extract the Clique engine to own crate
* Extract NullEngine and the block_reward module from ethcore
* Extract InstantSeal engine to own crate
* Extract remaining engines
* Extract executive_state to own crate so it can be used by engine crates
* Remove snapshot stuff from the engine crate
* Put snapshot traits back in ethcore
* cleanup
* Remove stuff from ethcore
* Don't use itertools
* itertools in aura is legit-ish
* More post-merge fixes
* Re-export less types in client
* cleanup
* Extract spec to own crate
* Put back the test-helpers from basic-authority
* Fix ethcore benchmarks
* Reduce the public api of ethcore/verification
* Update ethcore/block-reward/Cargo.toml
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update ethcore/engines/basic-authority/Cargo.toml
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update ethcore/engines/ethash/Cargo.toml
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* Update ethcore/engines/clique/src/lib.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* signers is already a ref
* Add an EngineType enum to tighten up Engine.name()
* Introduce Snapshotting enum to distinguish the type of snapshots a chain uses
* Rename supports_warp to snapshot_mode
* Missing import
* Update ethcore/src/snapshot/consensus/mod.rs
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* missing import
* Fix import
* double semi
* Temp storage for the private state added
* Temp storage for the private state added
* Request message added
* Store and retrieve offchain state logic
* State sync cache
* Private state column added to key value db
* Private state column added to key value db
* Indexing stored states via its hash
* Works with errors changed
* Private state stored into the local db
* Access to private state db added to sync io
* Private state db file added
* Rlp packets for retrieiving private state data added
* Handling of private sync completed message
* Test code fixed
* External flag for offchain storing added
* Test for private state sync added
* Saving private state logic corrected
* Migration code corrected
* Fixes after merge with master
* Merge with head
* Additional checks for slices
* Log for private state retrieval added
* Limit time of retrieving private states
* Store required hashes for every request, mark them stale if needed
* Store requested private state hashes and check received data
* Log stale requests
* State insertion fix
* Refactoring of how logging passed to state store
* Heapsize removed, syncing hashes structure reworked
* Check state length returned by contract
* Get rid of OverlayDB
* hash-db version updated
* Test fixed
* One more test fixed
* Add client-traits crate
Move the BlockInfo trait to new crate
* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.
* Use new machine and client-traits crates in ethcore
* Use new crates machine and client-traits instead of ethcore where appropriate
* Fix tests
* Don't re-export so many types from ethcore::client
* Fixing more fallout from removing re-export
* fix test
* More fallout from not re-exporting types
* Add some docs
* cleanup
* import the macro edition style
* Tweak docs
* Add missing import
* remove unused ethabi_derive imports
* Use latest ethabi-contract
* Move many traits from ethcore/client/traits to client-traits crate
Initial version of extracted Engine trait
* Move snapshot related traits to the engine crate (eew)
* Move a few snapshot related types to common_types
Cleanup Executed as exported from machine crate
* fix warning
* Gradually introduce new engine crate: snapshot
* ethcore typechecks with new engine crate
* Sort out types outside ethcore
* Add an EpochVerifier to ethash and use that in Engine.epoch_verifier()
Cleanup
* Document pub members
* Sort out tests
Sort out default impls for EpochVerifier
* Add test-helpers feature and move EngineSigner impl to the right place
* Sort out tests
* Sort out tests and refactor verification types
* Fix missing traits
* More missing traits
Fix Histogram
* Fix tests and cleanup
* cleanup
* Put back needed logger import
* Don't rexport common_types from ethcore/src/client
Don't export ethcore::client::*
* Remove files no longer used
Use types from the engine crate
Explicit exports from engine::engine
* Get rid of itertools
* Move a few more traits from ethcore to client-traits: BlockChainReset, ScheduleInfo, StateClient
* Move ProvingBlockChainClient to client-traits
* Don't re-export ForkChoice and Transition from ethcore
* Address grumbles: sort imports, remove commented out code
* Fix merge resolution error
* merge failure
* Add client-traits crate
Move the BlockInfo trait to new crate
* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.
* Use new machine and client-traits crates in ethcore
* Use new crates machine and client-traits instead of ethcore where appropriate
* Fix tests
* Don't re-export so many types from ethcore::client
* Fixing more fallout from removing re-export
* fix test
* More fallout from not re-exporting types
* Add some docs
* cleanup
* import the macro edition style
* Tweak docs
* Add missing import
* remove unused ethabi_derive imports
* Use latest ethabi-contract
* docs: Update Readme with TOC, Contributor Guide and License sections
* docs: Simplify package descriptions in Readme
* docs: Fix typos
* docs: Update Cargo package configs adding missing descriptions
* fix: Remove machine since was accidental from diff branch. Fix contribting indentation.
* fix: Fix formatting
* remove details snippet since not code and formats badly
* review-fix: Remove duplicate description
* fix dot point formatting
* fix: Add missing description to vm package
* fix: Remove duplicate Code of Conduct and Contributor guidelines
* docs: Update Contributing for consistency with other repos
* docs: Update Readme to link to existing Code of Conduct and Contributor guidelines in .github folder
* refactor: Add missing space
* fix: Fix links to be markdown format
* review-fix: Update CONTRIBUTING.md to link to Parity Ethereum Style Guide
* docs: Fix type in .github/CONTRIBUTING.md
Co-Authored-By: David <dvdplm@gmail.com>
* docs: Fix typo in .github/CONTRIBUTING.md
Co-Authored-By: David <dvdplm@gmail.com>
* docs: Rephrase parity-clib/Cargo.toml
Co-Authored-By: David <dvdplm@gmail.com>
* review-fix: Fix whole paragraph
* review-fix: Add comment to prevent direct pushes to master branch
* review-fix: Change some rules to recommendations
* Update .github/CONTRIBUTING.md
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* review-fix: Replace feature reduction with breaking changes dot point
* review-fix: Rephrase what to do when reviewing PR
* review-fix: Update parity-rpc package description and module rustdocs
* docs: Add missing fullstop
* review-fix: Update rustdoc section to show all packages first as default
* review-fix: Rename Parity Util to Parity Core Libraries
* review-fix: Rename readme too for the C bindings name change
* review-fix: Remove some docs since we do not teach the reader Rust
* review-fix: Wrap Parity Ethereum specific packages in a details section
* review-fix: Separate tools that are in this repo vs those that are not
* review-fix: Add link to ethabi on crates.io
* review-fix: Remove unnecessary extra line talking about pushing to master
* review-fix: Remove useless summary of parts of the rust book documentation section
* docs: Update .github/CONTRIBUTING.md to specify two reviewers required
Co-Authored-By: David <dvdplm@gmail.com>
* docs: Update README.md with simplification
Co-Authored-By: David <dvdplm@gmail.com>
* docs: Update README.md renaming to Test Runner instead of just Run
Co-Authored-By: David <dvdplm@gmail.com>
* review-fix: Remove hardware wallet from docs
* review-fix: Remove veto PR section of contributor doc
* review-fix: Change to test generator title
Define a new `enum` where devp2p subprotocol packet ids (currently eth and par) are defined. Additionally provide functionality to query id value and protocol of a given id object.
* Extract accounts from ethcore.
* Fix ethcore.
* Get rid of AccountProvider in test_helpers
* Fix rest of the code.
* Re-use EngineSigner, fix tests.
* Simplify EngineSigner to always have an Address.
* Fix RPC tests.
* Add deprecation notice to RPCs.
* Feature to disable accounts.
* extract accounts in RPC
* Run with accounts in tests.
* Fix RPC compilation and tests.
* Fix compilation of the binary.
* Fix compilation of the binary.
* Fix compilation with accounts enabled.
* Fix tests.
* Update submodule.
* Remove android.
* Use derive for Default
* Don't build secretstore by default.
* Add link to issue.
* Refresh Cargo.lock.
* Fix miner tests.
* Update rpc/Cargo.toml
Co-Authored-By: tomusdrw <tomusdrw@users.noreply.github.com>
* Fix private tests.
* Replace ethcore-logger with env-logger.
* Fix logger initialization in WASM tests.
* uncomment logger initialization in secret store
* Don't use ethcore-logger in whisper.
* Move ethcore-logger within parity dir.
* Uncomment rest from secret-store.
* Use `let _ =` in private_contract for consistency
* `ok()` to `let _ =` fix in service
* Use `let _ = ` for state_db
* Add a `fastmap` crate that provides the H256FastMap specialized HashMap
* Use `fastmap` instead of `plain_hasher`
* Update submodules for Reasons™
* Submodule update
* Bump parking_lot to 0.6
* Bump parity-wasm to 0.31 so it gets rid of parking_lot
ref https://github.com/paritytech/parity-wasm/pull/206
* Update jsonrpc versions
* Update wasmi and pwasm-utils version
* Fix compile
* Update jsonrpc crates
The `patricia_trie` crate is generic over the hasher (by way of HashDB) and node encoding scheme. Adds a new `patricia_trie_ethereum` crate with concrete impls for Keccak/RLP.
* Mark test helpers and test-only specs as cfg(test)
* Use test-probe to conditionally compile test helpers
* Remove test probe and directly use features tag
* Alter IO queueing.
* Don't require IoMessages to be Clone
* Ancient blocks imported via IoChannel.
* Get rid of private transactions io message.
* Get rid of deadlock and fix disconnected handler.
* Revert to old disconnect condition.
* Fix tests.
* Fix deadlock.
