Commit Graph

13 Commits

Author SHA1 Message Date
David
f6c3d4c695 Use upstream rocksdb (#11248)
* Use upstream rocksdb

…by way of https://github.com/paritytech/parity-common/pull/257 by @ordian.

* Hint at how `parity db reset` works in the error message

* migration-rocksdb: fix build

* Cargo.toml: use git dependency instead of path

* update to latest kvdb-rocksdb

* fix tests

* saner default for light client

* rename open_db to open_db_light

* update to latest kvdb-rocksdb

* moar update to latest kvdb-rocksdb

* even moar update to latest kvdb-rocksdb

* use kvdb-rocksdb from crates.io

* Update parity/db/rocksdb/helpers.rs

* add docs to memory_budget division
2019-12-03 16:59:11 +01:00
Andronik Ordian
df1c5ac616 ethcore: fix rlp deprecation warnings (#11280) 2019-11-21 17:23:49 +01:00
Jim Posen
c7219aae30
Upgrade trie-db to 0.16.0. (#11274)
* Upgrade trie-db to 0.16.0.

* Respond to review comments.
2019-11-21 11:08:49 +01:00
David
eb565a7e2b
Add a benchmark for snapshot::account::to_fat_rlps() (#11185)
* Add a benchmark for snapshot::account::to_fat_rlps()

`to_fat_rlps()` is a hot call during snapshots. I don't think it has a perf problem per se, but better to have benchmark for it.

The data used is a piece of Ropsten data sized to the ~95% percentile of account size on that network.

* Benchmark with more chunks, including mainnet data

* whitespace

* Move `used_code` inside the benchmark iteration

* Revert "Move `used_code` inside the benchmark iteration"

This reverts commit cff33ab30acbd1c009e745f646f1cc655ee01d8c.
2019-11-15 12:27:24 +01:00
David
8c2199dd2a
Snapshot restoration overhaul (#11219)
* Comments and todos
Use `snapshot_sync` as logging target

* fix compilation

* More todos, more logs

* Fix picking snapshot peer: prefer the one with the highest block number
More docs, comments, todos

* Adjust WAIT_PEERS_TIMEOUT to be a multiple of MAINTAIN_SYNC_TIMER to try to fix snapshot startup problems
Docs, todos, comments

* Tabs

* Formatting

* Don't build new rlp::EMPTY_LIST_RLP instances

* Dial down debug logging

* Don't warn about missing hashes in the manifest: it's normal
Log client version on peer connect

* Cleanup

* Do not skip snapshots further away than 30k block from the highest block seen

Currently we look for peers that seed snapshots that are close to the highest block seen on the network (where "close" means withing 30k blocks). When a node starts up we wait for some time (5sec, increased here to 10sec) to let peers connect and if we have found a suitable peer to sync a snapshot from at the end of that delay, we start the download; if none is found and --warp-barrier is used we stall, otherwise we start a slow-sync.
When looking for a suitable snapshot, we use the highest block seen on the network to check if a peer has a snapshot that is within 30k blocks of that highest block number. This means that in a situation where all available snapshots are older than that, we will often fail to start a snapshot at all. What's worse is that the longer we delay starting a snapshot sync (to let more peers connect, in the hope of finding a good snapshot), the more likely we are to have seen a high block and thus the more likely we become to accept a snapshot.
This commit removes this comparison with the highest blocknumber criteria entirely and picks the best snapshot we find in 10sec.

* lockfile

* Add a `ChunkType::Dupe` variant so that we do not disconnect a peer if they happen to send us a duplicate chunk (just ignore the chunk and keep going)
Resolve some documentation todos, add more

* tweak log message

* Don't warp sync twice
Check if our own block is beyond the given warp barrier (can happen after we've completed a warp sync but are not quite yet synced up to the tip) and if so, don't sync.
More docs, resolve todos.
Dial down some `sync` debug level logging to trace

* Avoid iterating over all snapshot block/state hashes to find the next work item

Use a HashSet instead of a Vec and remove items from the set as chunks are processed. Calculate and store the total number of chunks in the `Snapshot`  struct instead of counting pending chunks each time.

* Address review grumbles

* Log correct number of bytes written to disk

* Revert ChunkType::Dup change

* whitespace grumble

* Cleanup debugging code

* Fix docs

* Fix import and a typo

* Fix test impl

* Use `indexmap::IndexSet` to ensure chunk hashes are accessed in order

* Revert increased SNAPSHOT_MANIFEST_TIMEOUT: 5sec should be enough
2019-10-31 16:07:21 +01:00
David
0d3423cbe0
Use a lock instead of atomics for snapshot Progress (#11197)
* WIP. Typos and logging.

* Format todos

* Pause pruning while a snapshot is under way
Logs, docs and todos

* Allocate memory for the full chunk

* Name snapshotting threads

* Ensure `taking_snapshot` is set to false whenever and however `take_snapshot`returns
Rename `take_at` to `request_snapshot_at`
Cleanup

* Let "in_progress" deletion fail
Fix tests

* Just use an atomic

* Review grumbles

* Finish the sentence

* Resolve a few todos and clarify comments.

* Calculate progress rate since last update

* Lockfile

* Fix tests

* typo

* Reinstate default snapshotting frequency
Cut down on the logging noise

* Use a lock instead of atomics for snapshot Progress

* Update ethcore/types/src/snapshot.rs

Co-Authored-By: Andronik Ordian <write@reusable.software>

* Avoid truncating cast
Cleanup
2019-10-28 18:24:45 +01:00
David
ffeaee778c
Pause pruning while snapshotting (#11178)
* WIP. Typos and logging.

* Format todos

* Pause pruning while a snapshot is under way
Logs, docs and todos

* Allocate memory for the full chunk

* Name snapshotting threads

* Ensure `taking_snapshot` is set to false whenever and however `take_snapshot`returns
Rename `take_at` to `request_snapshot_at`
Cleanup

* Let "in_progress" deletion fail
Fix tests

* Just use an atomic

* Review grumbles

* Finish the sentence

* Resolve a few todos and clarify comments.

* Calculate progress rate since last update

* Lockfile

* Fix tests

* typo

* Reinstate default snapshotting frequency
Cut down on the logging noise

* address grumble

* Log memory use with `journal_size()` and explain why.
2019-10-24 16:46:32 +02:00
Anton Gavrilov
834585d61b
Crypto primitives removed from ethkey (#11174)
* Crypto utils removed from ethkey

* Fix ethkey lib

* Switch ethsore to new crypto

* Accounts crate fixed

* Secret store crate switched to new crypto

* Ethcore builtin fixed

* Accounts crate fixed

* Ethcore crate fixed

* Util network fixed

* Util network-devp2p fixed

* Private tx fixed

* Ethcore sync fixed

* Secret store fixed

* Rpc fixed

* Parity fixed

* Ethkey cli fixed

* Local store fixed

* Ethcore blockchain fixed

* Cargo.lock pushed; doc comment added for reversed nonce

* Ethstore tests fixed

* Ethstore cli fixed

* Miner fixed

* Snapshot tests are fixed

* Single brackets removed

* Machine fixed

* Verification fixed

* Executive state fixed

* More single brackets removed

* Update version of parity-crypto

* Use published version 0.4.2 of parity-crypto

* New test in tx_filter fixed
2019-10-23 13:03:46 +02:00
Niklas Adolfsson
2d2513b35a [dependencies] bump rand 0.7 (#11022)
* network-devp2p: bump rand `0.7`

* updater: bump rand `0.7`

* hash-fetch: bump rand `0.7`

* ethcore-sync: bump rand `0.7`

* rpc: dont work yet

* [private-tx] remove unused rand

* [ethcore-snapshot] bump rand 0.7

* [engine clique]: bump rand 0.7

* [engine authority-round]: remove rand

* [ethcore-blockchain]: bump rand 0.7

* [ethcore]: bump rand 0.7

* [ethcore-light]: bump rand 0.7

* [ethstore]: bump rand 0.7

* Fix for rand usage in rpc

* [rpc]: fix test build

* [ethcore-sync]: fix test build

* [snapshot tests]: rand 0.7

* [ethkey]: bump rand 0.7

* [rpc]: resolve TODO incompatible `rand versions`

* [ethkey] use `rust-secp256k1` master branch

* fix(bad merge): ethcoore-light remove itertools

* [rpc tests]: revert rpc test changes in #11139

`#11139` makes use a different `RNG/seed`, not `H64::random_using(&mut self.rand)`

This commit reverts the changed tests (the generated id by `Subscribers::next_id`)

* [ethkey/random]: resolve introduced `TODO`
2019-10-18 15:12:16 +02:00
Anton Gavrilov
f48780c29b
Update ethereum types to 0.8.0 version (#11139)
* Ethereum types updated to 0.8 version

* Fix for rand usage in rpc

* Cargo lock fixed after rebase

* TODO added
2019-10-08 14:18:44 +02:00
David
2b8b8851ab Update kvdb, kvdb-rocksdb and h2 (#11091)
* Update a few dependencies

Updates two dependencies: `kvdb-rocksdb` and `h2`. Brings in `parking_lot 0.9` which is unintended but possibly fine.

* Bump parking_lot to 0.9
Bump kvdb-memorydb to 0.2 (from git atm)

* New kvdb-memorydb is not breaking

* Remove [patch]
2019-09-27 23:29:14 +02:00
David
48629c2bd4
Move snapshot related traits to their proper place (#11012)
* Move snapshot to own crate
Sort out imports

* WIP cargo toml

* Make snapshotting generic over the client
Sort out tests

* Sort out types from blockchain and client

* Sort out sync

* Sort out imports and generics

* Sort out main binary

* Fix sync test-helpers

* Sort out import for secret-store

* Sort out more imports

* Fix easy todos

* cleanup

* Move SnapshotClient and SnapshotWriter to their proper places
Sort out the circular dependency between snapshot and ethcore by moving all snapshot tests to own crate, snapshot-tests

* cleanup

* Cleanup

* fix merge issues

* Update ethcore/snapshot/snapshot-tests/Cargo.toml

Co-Authored-By: Andronik Ordian <write@reusable.software>

* Sort out botched merge

* Ensure snapshot-tests run

* Docs

* Fix grumbles
2019-09-10 22:44:33 +02:00
David
d193ddde19
Extract snapshot to own crate (#11010)
* Move snapshot to own crate
Sort out imports

* WIP cargo toml

* Make snapshotting generic over the client
Sort out tests

* Sort out types from blockchain and client

* Sort out sync

* Sort out imports and generics

* Sort out main binary

* Fix sync test-helpers

* Sort out import for secret-store

* Sort out more imports

* Fix easy todos

* cleanup

* Cleanup

* remove unneded workspace member

* cleanup

* Sort out test-helpers dependency on account-db

* Update ethcore/client-traits/src/lib.rs

Co-Authored-By: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* Update ethcore/snapshot/Cargo.toml
2019-09-03 11:29:25 +02:00