Commit Graph

109 Commits

Author SHA1 Message Date
David
d193ddde19
Extract snapshot to own crate (#11010)
* Move snapshot to own crate
Sort out imports

* WIP cargo toml

* Make snapshotting generic over the client
Sort out tests

* Sort out types from blockchain and client

* Sort out sync

* Sort out imports and generics

* Sort out main binary

* Fix sync test-helpers

* Sort out import for secret-store

* Sort out more imports

* Fix easy todos

* cleanup

* Cleanup

* remove unneded workspace member

* cleanup

* Sort out test-helpers dependency on account-db

* Update ethcore/client-traits/src/lib.rs

Co-Authored-By: Niklas Adolfsson <niklasadolfsson1@gmail.com>

* Update ethcore/snapshot/Cargo.toml
2019-09-03 11:29:25 +02:00
David
cd26526868
Make ClientIoMessage generic over the Client (#10981)
* Add client-traits crate
Move the BlockInfo trait to new crate

* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.

* Use new machine and client-traits crates in ethcore

* Use new crates machine and client-traits instead of ethcore where appropriate

* Fix tests

* Don't re-export so many types from ethcore::client

* Fixing more fallout from removing re-export

* fix test

* More fallout from not re-exporting types

* Add some docs

* cleanup

* import the macro edition style

* Tweak docs

* Add missing import

* remove unused ethabi_derive imports

* Use latest ethabi-contract

* Move many traits from ethcore/client/traits to client-traits crate
Initial version of extracted Engine trait

* Move snapshot related traits to the engine crate (eew)

* Move a few snapshot related types to common_types
Cleanup Executed as exported from machine crate

* fix warning

* Gradually introduce new engine crate: snapshot

* ethcore typechecks with new engine crate

* Sort out types outside ethcore

* Add an EpochVerifier to ethash and use that in Engine.epoch_verifier()
Cleanup

* Document pub members

* Sort out tests
Sort out default impls for EpochVerifier

* Add test-helpers feature and move EngineSigner impl to the right place

* Sort out tests

* Sort out tests and refactor verification types

* Fix missing traits

* More missing traits
Fix Histogram

* Fix tests and cleanup

* cleanup

* Put back needed logger import

* Don't rexport common_types from ethcore/src/client
Don't export ethcore::client::*

* Remove files no longer used
Use types from the engine crate
Explicit exports from engine::engine

* Get rid of itertools

* Move a few more traits from ethcore to client-traits: BlockChainReset, ScheduleInfo, StateClient

* Move ProvingBlockChainClient to client-traits

* Don't re-export ForkChoice and Transition from ethcore

* Address grumbles: sort imports, remove commented out code

* Fix merge resolution error

* Extract the Clique engine to own crate

* Extract NullEngine and the block_reward module from ethcore

* Extract InstantSeal engine to own crate

* Extract remaining engines

* Extract executive_state to own crate so it can be used by engine crates

* Remove snapshot stuff from the engine crate

* Put snapshot traits back in ethcore

* cleanup

* Remove stuff from ethcore

* Don't use itertools

* itertools in aura is legit-ish

* More post-merge fixes

* Re-export less types in client

* cleanup

* Extract spec to own crate

* Put back the test-helpers from basic-authority

* Fix ethcore benchmarks

* Reduce the public api of ethcore/verification

* WIP

* Add Cargo.toml

* Fix compilation outside ethcore

* Audit uses of import_verified_blocks() and remove unneeded calls
Cleanup

* cleanup

* Remove unused imports from ethcore

* Cleanup

* remove double semi-colons

* Add missing generic param

* More missing generics

* Update ethcore/block-reward/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/basic-authority/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/ethash/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/clique/src/lib.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* signers is already a ref

* Add an EngineType enum to tighten up Engine.name()

* Introduce Snapshotting enum to distinguish the type of snapshots a chain uses

* Rename supports_warp to snapshot_mode

* Missing import

* Update ethcore/src/snapshot/consensus/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* missing import

* Fix import

* double semi

* Fix merge problem

* cleanup

* Parametrise `ClientIoMessage` with `()` for the light client

* Add impl Tick for ()

* Address review feedback

* Move ClientIoMessage to common-types

* remove superseeded fixme

* fix merge conflict errors
2019-08-28 10:09:42 +02:00
Niklas Adolfsson
dab2a6bd4b fix compile warnings (#10993)
* fix warnings

* fix: failing build, use `spec` as dev-dependency
2019-08-27 17:29:33 +02:00
David
fbf425c4e2 Extract spec to own crate (#10978)
* Add client-traits crate
Move the BlockInfo trait to new crate

* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.

* Use new machine and client-traits crates in ethcore

* Use new crates machine and client-traits instead of ethcore where appropriate

* Fix tests

* Don't re-export so many types from ethcore::client

* Fixing more fallout from removing re-export

* fix test

* More fallout from not re-exporting types

* Add some docs

* cleanup

* import the macro edition style

* Tweak docs

* Add missing import

* remove unused ethabi_derive imports

* Use latest ethabi-contract

* Move many traits from ethcore/client/traits to client-traits crate
Initial version of extracted Engine trait

* Move snapshot related traits to the engine crate (eew)

* Move a few snapshot related types to common_types
Cleanup Executed as exported from machine crate

* fix warning

* Gradually introduce new engine crate: snapshot

* ethcore typechecks with new engine crate

* Sort out types outside ethcore

* Add an EpochVerifier to ethash and use that in Engine.epoch_verifier()
Cleanup

* Document pub members

* Sort out tests
Sort out default impls for EpochVerifier

* Add test-helpers feature and move EngineSigner impl to the right place

* Sort out tests

* Sort out tests and refactor verification types

* Fix missing traits

* More missing traits
Fix Histogram

* Fix tests and cleanup

* cleanup

* Put back needed logger import

* Don't rexport common_types from ethcore/src/client
Don't export ethcore::client::*

* Remove files no longer used
Use types from the engine crate
Explicit exports from engine::engine

* Get rid of itertools

* Move a few more traits from ethcore to client-traits: BlockChainReset, ScheduleInfo, StateClient

* Move ProvingBlockChainClient to client-traits

* Don't re-export ForkChoice and Transition from ethcore

* Address grumbles: sort imports, remove commented out code

* Fix merge resolution error

* Extract the Clique engine to own crate

* Extract NullEngine and the block_reward module from ethcore

* Extract InstantSeal engine to own crate

* Extract remaining engines

* Extract executive_state to own crate so it can be used by engine crates

* Remove snapshot stuff from the engine crate

* Put snapshot traits back in ethcore

* cleanup

* Remove stuff from ethcore

* Don't use itertools

* itertools in aura is legit-ish

* More post-merge fixes

* Re-export less types in client

* cleanup

* Extract spec to own crate

* Put back the test-helpers from basic-authority

* Fix ethcore benchmarks

* Reduce the public api of ethcore/verification

* Update ethcore/block-reward/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/basic-authority/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/ethash/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/clique/src/lib.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* signers is already a ref

* Add an EngineType enum to tighten up Engine.name()

* Introduce Snapshotting enum to distinguish the type of snapshots a chain uses

* Rename supports_warp to snapshot_mode

* Missing import

* Update ethcore/src/snapshot/consensus/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* missing import

* Fix import

* double semi
2019-08-23 15:32:58 +02:00
David
2af3140a26
Extract engines to own crates (#10966)
* Add client-traits crate
Move the BlockInfo trait to new crate

* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.

* Use new machine and client-traits crates in ethcore

* Use new crates machine and client-traits instead of ethcore where appropriate

* Fix tests

* Don't re-export so many types from ethcore::client

* Fixing more fallout from removing re-export

* fix test

* More fallout from not re-exporting types

* Add some docs

* cleanup

* import the macro edition style

* Tweak docs

* Add missing import

* remove unused ethabi_derive imports

* Use latest ethabi-contract

* Move many traits from ethcore/client/traits to client-traits crate
Initial version of extracted Engine trait

* Move snapshot related traits to the engine crate (eew)

* Move a few snapshot related types to common_types
Cleanup Executed as exported from machine crate

* fix warning

* Gradually introduce new engine crate: snapshot

* ethcore typechecks with new engine crate

* Sort out types outside ethcore

* Add an EpochVerifier to ethash and use that in Engine.epoch_verifier()
Cleanup

* Document pub members

* Sort out tests
Sort out default impls for EpochVerifier

* Add test-helpers feature and move EngineSigner impl to the right place

* Sort out tests

* Sort out tests and refactor verification types

* Fix missing traits

* More missing traits
Fix Histogram

* Fix tests and cleanup

* cleanup

* Put back needed logger import

* Don't rexport common_types from ethcore/src/client
Don't export ethcore::client::*

* Remove files no longer used
Use types from the engine crate
Explicit exports from engine::engine

* Get rid of itertools

* Move a few more traits from ethcore to client-traits: BlockChainReset, ScheduleInfo, StateClient

* Move ProvingBlockChainClient to client-traits

* Don't re-export ForkChoice and Transition from ethcore

* Address grumbles: sort imports, remove commented out code

* Fix merge resolution error

* Extract the Clique engine to own crate

* Extract NullEngine and the block_reward module from ethcore

* Extract InstantSeal engine to own crate

* Extract remaining engines

* Extract executive_state to own crate so it can be used by engine crates

* Remove snapshot stuff from the engine crate

* Put snapshot traits back in ethcore

* cleanup

* Remove stuff from ethcore

* Don't use itertools

* itertools in aura is legit-ish

* More post-merge fixes

* Re-export less types in client

* cleanup

* Update ethcore/block-reward/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/basic-authority/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/ethash/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/clique/src/lib.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* signers is already a ref

* Add an EngineType enum to tighten up Engine.name()

* Introduce Snapshotting enum to distinguish the type of snapshots a chain uses

* Rename supports_warp to snapshot_mode

* Missing import

* Update ethcore/src/snapshot/consensus/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* remove double-semicolons
2019-08-22 18:25:49 +02:00
Anton Gavrilov
66e4410be7
Private contract migration and offchain state sync (#10748)
* Temp storage for the private state added

* Temp storage for the private state added

* Request message added

* Store and retrieve offchain state logic

* State sync cache

* Private state column added to key value db

* Private state column added to key value db

* Indexing stored states via its hash

* Works with errors changed

* Private state stored into the local db

* Access to private state db added to sync io

* Private state db file added

* Rlp packets for retrieiving private state data added

* Handling of private sync completed message

* Test code fixed

* External flag for offchain storing added

* Test for private state sync added

* Saving private state logic corrected

* Migration code corrected

* Fixes after merge with master

* Merge with head

* Additional checks for slices

* Log for private state retrieval added

* Limit time of retrieving private states

* Store required hashes for every request, mark them stale if needed

* Store requested private state hashes and check received data

* Log stale requests

* State insertion fix

* Refactoring of how logging passed to state store

* Heapsize removed, syncing hashes structure reworked

* Check state length returned by contract

* Get rid of OverlayDB

* hash-db version updated

* Test fixed

* One more test fixed
2019-08-16 14:45:52 +02:00
David
6a9de9b11e
Extract the Engine trait (#10958)
* Add client-traits crate
Move the BlockInfo trait to new crate

* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.

* Use new machine and client-traits crates in ethcore

* Use new crates machine and client-traits instead of ethcore where appropriate

* Fix tests

* Don't re-export so many types from ethcore::client

* Fixing more fallout from removing re-export

* fix test

* More fallout from not re-exporting types

* Add some docs

* cleanup

* import the macro edition style

* Tweak docs

* Add missing import

* remove unused ethabi_derive imports

* Use latest ethabi-contract

* Move many traits from ethcore/client/traits to client-traits crate
Initial version of extracted Engine trait

* Move snapshot related traits to the engine crate (eew)

* Move a few snapshot related types to common_types
Cleanup Executed as exported from machine crate

* fix warning

* Gradually introduce new engine crate: snapshot

* ethcore typechecks with new engine crate

* Sort out types outside ethcore

* Add an EpochVerifier to ethash and use that in Engine.epoch_verifier()
Cleanup

* Document pub members

* Sort out tests
Sort out default impls for EpochVerifier

* Add test-helpers feature and move EngineSigner impl to the right place

* Sort out tests

* Sort out tests and refactor verification types

* Fix missing traits

* More missing traits
Fix Histogram

* Fix tests and cleanup

* cleanup

* Put back needed logger import

* Don't rexport common_types from ethcore/src/client
Don't export ethcore::client::*

* Remove files no longer used
Use types from the engine crate
Explicit exports from engine::engine

* Get rid of itertools

* Move a few more traits from ethcore to client-traits: BlockChainReset, ScheduleInfo, StateClient

* Move ProvingBlockChainClient to client-traits

* Don't re-export ForkChoice and Transition from ethcore

* Address grumbles: sort imports, remove commented out code

* Fix merge resolution error

* merge failure
2019-08-15 17:59:22 +02:00
David
73f4564b66
Extract Machine from ethcore (#10949)
* Add client-traits crate
Move the BlockInfo trait to new crate

* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.

* Use new machine and client-traits crates in ethcore

* Use new crates machine and client-traits instead of ethcore where appropriate

* Fix tests

* Don't re-export so many types from ethcore::client

* Fixing more fallout from removing re-export

* fix test

* More fallout from not re-exporting types

* Add some docs

* cleanup

* import the macro edition style

* Tweak docs

* Add missing import

* remove unused ethabi_derive imports

* Use latest ethabi-contract
2019-08-13 12:33:34 +02:00
Marek Kotewicz
45978bc2bd unify loading spec && further spec cleanups (#10948)
* bundle_* macros for spec

* fixed failing tests

* renamed bundle.rs to chain.rs

* removed unused derives

* updated spec/chain.rs description
2019-08-07 16:52:48 +02:00
David
1ef9d5b52f Move more types out of ethcore (#10880)
* WIP move errors, pod_account and state account to own crates

* Sort out dependencies, fix broken code and tests
Remove botched ethcore-error crate

* remove template line

* fix review feedback

* Remove test-only AccountDBMut::new

* Extract AccountDB to account-db

* Move Substate to state-account – wip

* Add lib.rs

* cleanup

* test failure

* test failure 2

* third time's the charm

* Add factories crate

* Use new factories crate

* Use factories crate

* Extract trace

* Fix tests

* Sort out parity-util-mem and parking_lot

* cleanup

* WIP port over the rest of state from ethcore

* Collect all impls for Machine

* some notes

* Rename pod-account to pod

* Move PodState to pod crate

* Use PodState from pod crate

* Fix use clause for json tests

* Sort out evmbin

* Add missing code and use PodState

* Move code that depends on Machine and Executive to own module

* Sort out cloning errors, fix ethcore to use new state crate

* Do without funky From impls

* Fix ethcore tests

* Fixes around the project to use new state crate

* Add back the more specific impls of StateOrBlock From conversions

* Move execute to freestanding function and remove it from trait
Sort out the error handling in executive_state by moving the result types from state to ethcore
Undo the verbose code added to work around the StateOrBlock From conversions

* cleanup

* Fix "error: enum variants on type aliases are experimental"

* Bring back the state tests
Fix whitespace

* remove ethcore/state/mod.rs

* cleanup

* cleanup

* Cleanup state-account errors

* Fix more todos
Add module docs

* Add error.rs

* Fixup Cargo.lock

* Smaller ethcore API is fine

* Add `to-pod-full` feature to state-account
Fix evmbin

* Fix a few more test failures

* Fix RPC test build

* Baptize the new trait

* Remove resolved TODOs

* Rename state-account to account-state

* Do not re-export the trace crate

* Don't export state_db from ethcore

* Let private-tx use StateDB. :(

* Remove ethcore/src/pod_state.rs

* Inner type does not need to be pub/pub(crate)

* optimise imports

* Revert "Inner type does not need to be pub/pub(crate)"

This reverts commit 2f839f8a0f72f71334da64620f57e6dd6039f06b.

* Move DatabaseExtras to ethcore-blockchain

* Add database_extra module to ethcore-blockchain

* Remove to-pod-full feature

* cosmetics

* New crate: state-db

* Add new crate

* Move PreverifiedBlock and BlockError to types

* Sort out the merge

* Add missing `license` meta data keys

* wip

* wip client-traits

* merge conflict

* verification crate type checks

* Move impls for CommonParams to common_types
Fix misc stuff in ethcore

* Fix tests

* Implement VerifyingEngine for all engines except Ethash
Temporarily sort out error handling
Move more types to common_types

* Split Engine in two and move code around

* cleanup

* verification: don't rexport common_types

* Use error from common_types

* Consolidate error types

* VerifyingEngine use Errors from common_types

* verification: Use error type from common_types

* SnapshotError moved to common_types

* Move more code from Engne to VerifyingEngine
Add a VerifyingClient trait: BlockInfo + CallContract
Whitespace

* Add MAX_UNCLE_AGE const

* Port over remaining code from ethcore/verification

* Use errors from common_types

* Fix the confusing "io" naming

* Move more types into common_types

* Add todos

* Experiment with Engine trait outside ethcore

* Hook up types from common_types in ethcore
Don't use verification crate
Don't use client-traits crate

* Revert to impl Engine for Arc<Ethash> and add note to explain why
Revert moving ClientIoMessage to common_types
Fix build

* Remove ClientIoMessage from common_types

* Cleanup

* More cleanup

* Sort error handling changes in the rest of parity

* Remove unused code

* Remove WIP types

* Cleanup todos not tackled here

* remove cruft

* Fix some whitespace and a merge error

* ethcore tests

* test failures

* Restore Engine impls to master to make review a bit easier

* cleanup

* whitespace

* applied review suggestions

* types does not depend on rustc-hex

* ethash engine moved to engine module

* applied review suggestion
2019-07-18 12:27:08 +02:00
Marek Kotewicz
cfe826fae9
whisper is no longer a part of parity-ethereum repo (#10855)
* whisper is no longer a part of parity-ethereum repo

* fix failing tests

* update whisper help with the link to the new repo

* Removed AttachedProtocols

* updated whisper info in README.md files

* Update parity/deprecated.rs

Co-Authored-By: Andronik Ordian <write@reusable.software>
2019-07-12 14:15:30 +02:00
David
f53c3e582c Run cargo fix on a few of the worst offenders (#10854)
* Run cargo fix on `vm`

* Run cargo fix on ethcore-db

* Run cargo fix on evm

* Run cargo fix on ethcore-light

* Run cargo fix on journaldb

* Run cargo fix on wasm

* Missing docs

* Run cargo fix on ethcore-sync
2019-07-09 09:04:20 +01:00
David
44cc442d12
Move more code into state-account (#10840)
* WIP move errors, pod_account and state account to own crates

* Sort out dependencies, fix broken code and tests
Remove botched ethcore-error crate

* remove template line

* fix review feedback

* Remove test-only AccountDBMut::new

* Extract AccountDB to account-db

* Move Substate to state-account – wip

* Add lib.rs

* cleanup

* test failure

* test failure 2

* third time's the charm

* Add factories crate

* Use new factories crate

* Use factories crate

* Extract trace

* Fix tests

* Sort out parity-util-mem and parking_lot

* cleanup

* WIP port over the rest of state from ethcore

* Collect all impls for Machine

* some notes

* Rename pod-account to pod

* Move PodState to pod crate

* Use PodState from pod crate

* Fix use clause for json tests

* Sort out evmbin

* Add missing code and use PodState

* Move code that depends on Machine and Executive to own module

* Sort out cloning errors, fix ethcore to use new state crate

* Do without funky From impls

* Fix ethcore tests

* Fixes around the project to use new state crate

* Add back the more specific impls of StateOrBlock From conversions

* Move execute to freestanding function and remove it from trait
Sort out the error handling in executive_state by moving the result types from state to ethcore
Undo the verbose code added to work around the StateOrBlock From conversions

* cleanup

* Fix "error: enum variants on type aliases are experimental"

* Bring back the state tests
Fix whitespace

* remove ethcore/state/mod.rs

* cleanup

* cleanup

* Cleanup state-account errors

* Fix more todos
Add module docs

* Add error.rs

* Fixup Cargo.lock

* Smaller ethcore API is fine

* Add `to-pod-full` feature to state-account
Fix evmbin

* Fix a few more test failures

* Fix RPC test build

* Baptize the new trait

* Remove resolved TODOs

* Rename state-account to account-state

* Do not re-export the trace crate

* Don't export state_db from ethcore

* Let private-tx use StateDB. :(

* Remove ethcore/src/pod_state.rs

* Inner type does not need to be pub/pub(crate)

* optimise imports

* Revert "Inner type does not need to be pub/pub(crate)"

This reverts commit 2f839f8a0f72f71334da64620f57e6dd6039f06b.

* Move DatabaseExtras to ethcore-blockchain

* Add database_extra module to ethcore-blockchain

* Remove to-pod-full feature

* Sort out the merge

* sort imports

* address grumbles

* rename crate

* address more grumbles
2019-07-08 18:17:48 +02:00
Marek Kotewicz
c4c5d79a0f removed QueueError type (#10852) 2019-07-06 20:40:56 +02:00
David
de906d4afd
Improve logging and cleanup in miner around block sealing (#10745)
* Stop breaking out of loop if a non-canonical hash is found

* include expected hash in log msg

* More logging

* Scope

* Syntax

* Log in blank RollingFinality
Escalate bad proposer to warning

* Check validator set size: warn if 1 or even number

* More readable code

* Use SimpleList::new

* Extensive logging on unexpected non-canonical hash

* Wording

* wip

* Update ethcore/blockchain/src/blockchain.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Improved logging, address grumbles

* Update ethcore/src/engines/validator_set/simple_list.rs

Co-Authored-By: Luke Schoen <ltfschoen@users.noreply.github.com>

* Report benign misbehaviour iff currently a validator

* Report malicious behaviour iff we're a validator

* Escalate to warning and fix wording

* Test reporting behaviour
Don't require node to be part of the validator set to report malicious behaviour

* Include missing parent hash in MissingParent error

* Update ethcore/src/engines/validator_set/simple_list.rs

Co-Authored-By: Luke Schoen <ltfschoen@users.noreply.github.com>

* docs

* remove unneeded into()
Move check for parent_step == step for clarity&efficiency
Remove dead code for Seal::Proposal

* typo

* Wording

* naming

* WIP

* cleanup

* cosmetics

* cosmetics and one less lvar

* spelling

* Better loggin when a block is already in chain

* More logging

* On second thought non-validators are allowed to report

* cleanup

* remove dead code

* Keep track of the hash of the last imported block

* Let it lock

* Serialize access to block sealing

* Take a lock while sealing a block

* Cleanup

* whitespace
2019-07-04 18:03:22 +02:00
David
5dc5be1e58
Better logging when backfilling ancient blocks fail (#10796)
* Better logging when backfilling ancient blocks fail
Print total blocks imported, closes #10792

* `finalize()` doesn't need Engine
Pull out call to migrated_blocks() from replace_client_db()

* More logs

* Clarify that the percentage may be misleading

* Remove replace_client_db() and replace with a straight call to restore_db()

* Include the parent_hash in UnlinkedAncientBlockChain errors

* Add a new RestorationStatus varian: Finalizing (as it can take a loooong while)
Call abort_restore() when restoration fails

* Add missing cases for new variant

* typos

* Typo and derive Debug

* Do not attempt to salvage existing blocks unless they form a complete chain back to genesis

* Fix test

* Revert "Fix test"

This reverts commit f027d4b4cb7b6c23fceec528c1711886ba9cfe4e.

* Fix test again

* Update comment

* Be careful about locks

* fix test failure

* Do not defer returning an error when the chain is broken

* Review feedback

* no hex formatting for Option
2019-07-01 14:41:45 +02:00
Anton Gavrilov
59f0eb7e6b Do not drop the peer with None difficulty (#10772)
* Treat peer with None difficulty as legit

* Temporarily enable release binary build - REMOVE BEFORE MERGE
2019-06-27 17:45:21 +02:00
cheme
6fc5014b4d
Remove calls to heapsize (#10432)
* update memorydb trait
* use malloc_size_of instead of heapsize_of
* use jemalloc as default allocator for parity client.
2019-06-19 13:54:05 +02:00
David
d2120ded56 Add a way to signal shutdown to snapshotting threads (#10744)
* Add a way to signal shutdown to snapshotting threads

* Pass Progress to fat_rlps() so we can abort from there too.

* Checking for abort in a single spot

* Remove nightly-only weak/strong counts

* fix warning

* Fix tests

* Add dummy impl to abort snapshots

* Add another dummy impl for TestSnapshotService

* Remove debugging code

* Return error instead of the odd Ok(())
Switch to AtomicU64
2019-06-19 10:13:09 +02:00
Andrew Jones
bf55db4c7e Die error_chain, die (#10747)
* Replace error chain for network error

* Fix usages and add manual From impls

* OnDemand Error and remove remaining dependencies

* Die error_chain, die.

* DIE

* Hasta la vista, baby
2019-06-17 08:44:59 +02:00
Andronik Ordian
dae5d75dd6 Upgrade ethereum types (#10670)
* cargo upgrade "ethereum-types" --all --allow-prerelease

* [ethash] fix compilation errors

* [ethkey] fix compilation errors

* [journaldb] fix compilation errors

* [dir] fix compilation errors

* [ethabi] update to 0.7

* wip

* [eip-712] fix compilation errors

* [ethjson] fix compilation errors

* [Cargo.toml] add TODO to remove patches

* [ethstore] fix compilation errors

* use patched keccak-hash with new primitive-types

* wip

* [ethcore-network-devp2p] fix compilation errors

* [vm] fix compilation errors

* [common-types, evm, wasm] fix compilation errors

* [ethcore-db] Require AsRef instead of Deref for keys

* [ethcore-blockchain] fix some compilation errors

* [blooms-db] fix compilation errors

Thanks a lot @dvdplm :)

* we don't need no rlp ethereum feature

* [ethcore] fix some compilation errors

* [parity-ipfs-api] fix compilation error

* [ethcore-light] fix compilation errors

* [Cargo.lock] update parity-common

* [ethcore-private-tx] fix some compilation errors

* wip

* [ethcore-private-tx] fix compilation errors

* [parity-updater] fix compilation errors

* [parity-rpc] fix compilation errors

* [parity-bin] fix other compilation errors

* update to new ethereum-types

* update keccak-hash

* [fastmap] fix compilation in tests

* [blooms-db] fix compilation in tests

* [common-types] fix compilation in tests

* [triehash-ethereum] fix compilation in tests

* [ethkey] fix compilation in tests

* [pwasm-run-test] fix compilation errors

* [wasm] fix compilation errors

* [ethjson] fix compilation in tests

* [eip-712] fix compilation in tests

* [ethcore-blockchain] fix compilation in tests

* [ethstore] fix compilation in tests

* [ethstore-accounts] fix compilation in tests

* [parity-hash-fetch] fix compilation in tests

* [parity-whisper] fix compilation in tests

* [ethcore-miner] fix compilation in tests

* [ethcore-network-devp2p] fix compilation in tests

* [*] upgrade rand to 0.6

* [evm] get rid of num-bigint conversions

* [ethcore] downgrade trie-standardmap and criterion

* [ethcore] fix some warnings

* [ethcore] fix compilation in tests

* [evmbin] fix compilation in tests

* [updater] fix compilation in tests

* [ethash] fix compilation in tests

* [ethcore-secretstore] fix compilation in tests

* [ethcore-sync] fix compilation in tests

* [parity-rpc] fix compilation in tests

* [ethcore] finally fix compilation in tests

FUCK YEAH!!!

* [ethstore] lazy_static is unused

* [ethcore] fix test

* fix up bad merge

* [Cargo.toml] remove unused patches

* [*] replace some git dependencies with crates.io

* [Cargo.toml] remove unused lazy_static

* [*] clean up

* [ethcore] fix transaction_filter_deprecated test

* [private-tx] fix serialization tests

* fix more serialization tests

* [ethkey] fix smoky test

* [rpc] fix tests, please?

* [ethcore] remove commented out code

* Apply suggestions from code review

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* [ethstore] remove unused dev-dependency

* [ethcore] remove resolved TODO

* [*] resolve keccak-hash TODO

* [*] s/Address::default()/Address::zero()

* [rpc] remove Subscribers::new_test

* [rpc] remove EthPubSubClient::new_test

* [ethcore] use trie-standardmap from crates.io

* [dir] fix db_root_path

* [ethcore] simplify snapshot::tests::helpers::fill_storage

* Apply suggestions from code review

Co-Authored-By: David <dvdplm@gmail.com>

* [ethcore-secretstore] resolve TODO in serialization

* [ethcore-network-devp2p] resolve TODO in save_key

* [Cargo.lock] update triehash

* [*] use ethabi from crates.io

* [ethkey] use secp256k1 from master branch

* [Cargo.lock] update eth-secp256k1
2019-06-03 15:36:21 +02:00
David
c31ffab22e Remove annoying compiler warnings (#10679) 2019-05-21 16:12:49 +01:00
Anton Gavrilov
5a581c1c90 Trivial journal for private transactions (#10056)
* Journal for private txs added

* Tests after adding logging to private tx fixed

* Logs getter and tests added

* Time and amount limit for logs added

* RPC method for log retrieving added

* Correct path name and time validation implemented

* References for parameters added, redundant cloning reworked

* References for parameters added, redundant cloning reworked

* Work with json moved to the separate struct

* Serialization test added

* Fixed build after the merge with head

* Documentation for methods fixed, redundant field removed

* Fixed error usages

* Timestamp trait implemented for std struct

* Commented code removed

* Remove timestamp source, rework serialization test

* u64 replaced with SystemTime

* Path made mandatory for logging

* Source of monotonic time added

* into_system_time method renamed

* Initialize time source by max from current system time and max creation time from already saved logs

* Redundant conversions removed, code a little bit reworked according to review comments

* One more redundant conversion removed, rpc call simplified
2019-05-14 11:21:21 +02:00
Andrew Jones
98b89c8e4f [ethcore] remove error_chain (#10616)
* Derive Display for BlockError

* Convert error_chain errors

* Convert ethcore usages of errors

* Fix remaining compile errors in ethcore

* Fix other crates

* Fix tests compilation

* Implement error for Snapshot error

* Remove redundant into
2019-05-06 15:06:20 +02:00
Seun LanLege
fba63de974 RPC: Implements eth_subscribe("syncing") (#10311)
* implement eth_subscibe syncing

* fix imports

* add is_major_syncing to SyncProvider

* add eth_subscribe(syncing) support for light clients

* tests

* fix TestSyncClient

* correct LightFetch impl

* added currentBlock, startingBlock, highestBlock to PubSubSyncStatus

* fixed tests

* fix PR grumbles

* improve code style

* is_syncing -> syncing

* code style

* use ethereum types
2019-04-02 17:13:55 +02:00
elferdo
3adb640d2b
Bundle protocol and packet_id together in chain sync (#10315)
Define a new `enum` where devp2p subprotocol packet ids (currently eth and par) are defined. Additionally provide functionality to query id value and protocol of a given id object.
2019-02-13 09:20:33 +01:00
Niklas Adolfsson
751d15e4be fix(light-rpc): Make light_sync generic (#10238)
* fix(light-rpc): Make `light_sync` generic

The motivation behind this change is to easily mock `light-sync` to make it possible to enable `rpc-integration` tests
for the light-client.

Currently the `rpc's` requires the concrete type `sync::LightSync` which makes it very hard to do so

* fix(bad merge)
2019-02-11 10:33:16 +00:00
elferdo
046b8bbc8a
Make specification of protocol in SyncRequester::send_request explicit (#10295)
* Make the specification of the protocol to which a packet_id belongs to
  explicit when calling "SyncRequester::send_packet".

* Remove "SyncIO::send" and leave only "SyncIO::send_protocol"

* Adapt tests to new code.

* Strengthen tests to check if packet_id and protocol match when sending
a devp2p packet.
2019-02-08 13:01:29 +01:00
elferdo
b7e8621846 Increase number of requested block bodies in chain sync (#10247)
* Increase the number of block bodies requested during Sync.

* Increase the number of block bodies requested during Sync.

* Check if our peer is an older parity client with the bug
  of not handling large requests properly

* Add a ClientVersion struct and a ClientCapabilites trait

* Make ClientVersion its own module

* Refactor and extend use of ClientVersion

* Replace strings with ClientVersion in PeerInfo

* Group further functionality in ClientCapabilities

* Move parity client version data from tuple to its own struct.

* Implement accessor methods for ParityClientData and remove them
from ClientVersion.

* Minor fixes

* Make functions specific to parity return types specific to parity.

* Test for shorter ID strings

* Fix formatting and remove unneeded dependencies.

* Roll back Cargo.lock

* Commit last Cargo.lock

* Convert from string to ClientVersion

* * When checking if peer accepts service transactions just check
  if it's parity, remove version check.

* Remove dependency on semver in ethcore-sync

* Remove unnecessary String instantiation

* Rename peer_info to peer_version

* Update RPC test helpers

* Simplify From<String>

* Parse static version string only once

* Update RPC tests to new ClientVersion struct

* Document public members

* More robust parsing of ID string

* Minor changes.

* Update version in which large block bodies requests appear.

* Update ethcore/sync/src/block_sync.rs

Co-Authored-By: elferdo <elferdo@gmail.com>

* Update util/network/src/client_version.rs

Co-Authored-By: elferdo <elferdo@gmail.com>

* Update util/network/src/client_version.rs

Co-Authored-By: elferdo <elferdo@gmail.com>

* Update tests.

* Minor fixes.
2019-02-07 15:27:09 +01:00
Tomasz Drwięga
d5c19f8719 Deprecate account management (#10213)
* Extract accounts from ethcore.

* Fix ethcore.

* Get rid of AccountProvider in test_helpers

* Fix rest of the code.

* Re-use EngineSigner, fix tests.

* Simplify EngineSigner to always have an Address.

* Fix RPC tests.

* Add deprecation notice to RPCs.

* Feature to disable accounts.

* extract accounts in RPC

* Run with accounts in tests.

* Fix RPC compilation and tests.

* Fix compilation of the binary.

* Fix compilation of the binary.

* Fix compilation with accounts enabled.

* Fix tests.

* Update submodule.

* Remove android.

* Use derive for Default

* Don't build secretstore by default.

* Add link to issue.

* Refresh Cargo.lock.

* Fix miner tests.

* Update rpc/Cargo.toml

Co-Authored-By: tomusdrw <tomusdrw@users.noreply.github.com>

* Fix private tests.
2019-02-07 14:34:24 +01:00
Anton Gavrilov
45d7c60608 Call private contract methods from another private contract (read-only) (#10086)
* Patch available private contracts during private transaction

* Key acl ABI added

* Work with secret store keys moved to the separate struct

* Private tx test refactored

* Revert "Private tx test refactored"

This reverts commit 476c132d692c7a886bc7b7cd7fe47b3d7692bd63.

* Test for calling private contract from another one added

* Test fixed

* Redundant tab removed

* ACL contract processing fixed, test added

* Merge with head

* Expect replaced with closure
2019-02-07 12:39:04 +01:00
Niklas Adolfsson
4b11d79829
fix(ManageNetwork): replace Range with RangeInclusive (#10209)
* fix(ManageNetwork): replace Range -> RangeIncls

Fixes `TODO: Range should be changed to RangeInclusive once stable (https://github.com/rust-lang/rust/pull/50758)`

* fix(tests)

* fix(grumbles): off-by-one error in debug_asserts

* RangeInclusive::end() is inclusive which means that if start and end is equal the `debug_assert(range.end() >
range.start()` will fail which is shouldn't
2019-01-22 09:51:40 +01:00
Tomasz Drwięga
ab22d5e278 Replace ethcore-logger with env-logger. (#10102)
* Replace ethcore-logger with env-logger.

* Fix logger initialization in WASM tests.

* uncomment logger initialization in secret store

* Don't use ethcore-logger in whisper.

* Move ethcore-logger within parity dir.

* Uncomment rest from secret-store.

* Use `let _ =` in private_contract for consistency

* `ok()` to `let _ =` fix in service

* Use `let _ = ` for state_db
2019-01-08 15:07:20 +01:00
Afri Schoedon
7c335e8764
misc: bump license header to 2019 (#10135)
* misc: bump license header to 2019

* misc: remove_duplicate_empty_lines.sh

* misc: run license header script

* commit cargo lock
2019-01-07 11:33:07 +01:00
Nicolas Gotchac
e435407080 Don't try to send oversized packets (#10042)
* Don't construct oversized packets

* Add test for payload limit

* [eth-sync] Fix wrongly computed data sizes

* Replace `MAX_RECEIPTS_TO_SEND` with overall softlimit
2019-01-04 19:58:21 +01:00
Anton Gavrilov
b180be7526 Private tx enabled flag added into STATUS packet (#9999)
* Add private tx enabled flag into status packet

* Error log added for the case with no peers available

* Add flag only for supported protocol versions

* Work with private handler refactored

* Log target changed

* Cargo.lock updated
2019-01-04 19:57:01 +01:00
Tomasz Drwięga
3650f2d51c Extract blockchain from ethcore (#10114)
* Split blockchain & db from ethcore.

* Clean up blockchain deps.

* Missing docs.

* Fix blockchain tests.

* Make other crates compile.

* Remove some re-exports.

* Remove types re-export from ethcore.

* Remove EVM dependency from transaction.

* Merge ethcore-transaction with common-types.

* Clean-up ethcore deps a bit.

* remove ethcore from cargo.toml

* Update ethcore/blockchain/src/lib.rs

Co-Authored-By: tomusdrw <tomusdrw@users.noreply.github.com>

* Address review comments.

* Update DB comment.

* Add tracking issue to the TODO and fix typo.

* Common naming for common types.

* Update ethcore/db/src/keys.rs

Co-Authored-By: tomusdrw <tomusdrw@users.noreply.github.com>

* Update ethcore/blockchain/src/generator.rs

Co-Authored-By: tomusdrw <tomusdrw@users.noreply.github.com>

* Try to fix beta tests.
2019-01-04 14:05:46 +01:00
joshua-mir
130901e820 Identity fix (#10128)
* fix #10125

fix service transaction version detection if --identity is enabled, change test to match how --identity actually works

* fix wrong var

* get the index of  v, not /

* idx, not idx.len()

* Update ethcore/sync/src/chain/propagator.rs

Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>

* Update ethcore/sync/src/chain/propagator.rs

Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>

* change version prefix to a const

* space

Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>
2019-01-04 13:23:46 +01:00
mattrutherford
215602de08
Fix pubsub new_blocks notifications to include all blocks (#9987)
Fix: new blocks notifications sometimes missing in pubsub RPC

Implement new struct to pass to `new_blocks()` with extra parameter - `has_more_blocks_to_import`, which was previously used to determine whether the notification should be sent. Now it's up to each implementation to decide what to do.

Updated all implementations to behave as before, except `eth_pubsub`, which will send notification even when the queue is not empty.

Update tests.
2018-12-19 09:24:14 +00:00
Wei Tang
f092c10de5
Remove tendermint engine support (#9980)
* Remove tendermint engine support

* Remove tendermint test json spec

* Fix ethcore test compile

* Remove tendermint test in sync
2018-11-29 06:47:11 +08:00
Nicolas Gotchac
5e9dc185a5 Fix unstable peers and slowness in sync (#9967)
* Don't sync all peers after each response

* Update formating

* Fix tests: add `continue_sync` to `Sync_step`

* Update ethcore/sync/src/chain/mod.rs

Co-Authored-By: ngotchac <ngotchac@gmail.com>
2018-11-28 12:19:35 +01:00
Tomasz Drwięga
0b5bbf6048 Improve block and transaction propagation (#9954)
* Refactor sync to add priority tasks.

* Send priority tasks notifications.

* Propagate blocks, optimize transactions.

* Implement transaction propagation. Use sync_channel.

* Tone down info.

* Prevent deadlock by not waiting forever for sync lock.

* Fix lock order.

* Don't use sync_channel to prevent deadlocks.

* Fix tests.
2018-11-28 11:30:05 +01:00
Nicolas Gotchac
0d593199d0 Fix light client informant while syncing (#9932)
* Add `is_idle` to LightSync to check importing status

* Use SyncStateWrapper to make sure is_idle gets updates

* Update is_major_import to use verified queue size as well

* Add comment for `is_idle`

* Add Debug to `SyncStateWrapper`

* `fn get` -> `fn into_inner`
2018-11-26 14:05:02 +03:00
Nicolas Gotchac
664bb2becd Adjust requests costs for light client (#9925)
* PIP Table Cost relative to average peers instead of max peers

* Add tracing in PIP new_cost_table

* Update stat peer_count

* Use number of leeching peers for Light serve costs

* Fix test::light_params_load_share_depends_on_max_peers (wrong type)

* Remove (now) useless test

* Remove `load_share` from LightParams.Config
Prevent div. by 0

* Add LEECHER_COUNT_FACTOR

* PR Grumble: u64 to u32 for f64 casting

* Prevent u32 overflow for avg_peer_count

* Add tests for LightSync::Statistics
2018-11-21 20:11:01 +01:00
Nicolas Gotchac
9475a2e474 Keep existing blocks when restoring a Snapshot (#8643)
* Rename db_restore => client

* First step: make it compile!

* Second step: working implementation!

* Refactoring

* Fix tests

* PR Grumbles

* PR Grumbles WIP

* Migrate ancient blocks interating backward

* Early return in block migration if snapshot is aborted

* Remove RwLock getter (PR Grumble I)

* Remove dependency on `Client`: only used Traits

* Add test for recovering aborted snapshot recovery

* Add test for migrating old blocks

* Fix build

* PR Grumble I

* PR Grumble II

* PR Grumble III

* PR Grumble IV

* PR Grumble V

* PR Grumble VI

* Fix one test

* Fix test

* PR Grumble

* PR Grumbles

* PR Grumbles II

* Fix tests

* Release RwLock earlier

* Revert Cargo.lock

* Update _update ancient block_ logic: set local in `commit`

* Update typo in ethcore/src/snapshot/service.rs

Co-Authored-By: ngotchac <ngotchac@gmail.com>
2018-11-18 00:06:34 +01:00
Andrew Jones
9a2c4a34ee
sync: retry different peer after empty subchain heads response (#9753)
* If no subchain heads then try a different peer

* Add log when useless chain head

* Restrict ChainHead useless peer to ancient blocks

* sync: replace `limit_reorg` with `block_set` condition
2018-10-25 16:56:59 +02:00
David
c313039526 Update a few parity-common dependencies (#9663)
* Update a few parity-common dependencies

* cleanup

* cleanup

* revert update of ethereum/tests

* better reporting of network rlp errors

* Use rlp 0.3.0-beta.1

* fix util function get_dummy_blocks

* Already a Vec

* encode_list returns vec already

* Address grumble

* No need for betas

* Fix double spaces
2018-10-09 22:07:25 +02:00
Andrew Jones
4b6ebcbb61 Fix ancient blocks sync (#9531)
* Log block set in block_sync for easier debugging

* logging macros

* Match no args in sync logging macros

* Add QueueFull error

* Only allow importing headers if the first matches requested

* WIP

* Test for chain head gaps and log

* Calc distance even with 2 heads

* Revert previous commits, preparing simple fix

This reverts commit 5f38aa885b22ebb0e3a1d60120cea69f9f322628.

* Reject headers with no gaps when ChainHead

* Reset block sync download when queue full

* Simplify check for subchain heads

* Add comment to explain subchain heads filter

* Fix is_subchain_heads check and comment

* Prevent premature round completion after restart

This is a problem on mainnet where multiple stale peer requests will
force many rounds to complete quickly, forcing the retraction.

* Reset stale old blocks request after queue full

* Revert "Reject headers with no gaps when ChainHead"

This reverts commit 0eb865539e5dee37ab34f168f5fb643300de5ace.

* Add BlockSet to BlockDownloader logging

Currently it is difficult to debug this because there are two instances,
one for OldBlocks and one for NewBlocks. This adds the BlockSet to all
log messages for easy log filtering.

* Reset OldBlocks download from last enqueued

Previously when the ancient block queue was full it would restart the
download from the last imported block, so the ones still in the queue would be
redownloaded. Keeping the existing downloader instance and just
resetting it will start again from the last enqueued block.:wq

* Ignore expired Body and Receipt requests

* Log when ancient block download being restarted

* Only request old blocks from peers with >= difficulty

https://github.com/paritytech/parity-ethereum/pull/9226 might be too
permissive and causing the behaviour of the retraction soon after the
fork block. With this change the peer difficulty has to be greater than
or euqal to our syncing difficulty, so should still fix
https://github.com/paritytech/parity-ethereum/issues/9225

* Some logging and clear stalled blocks head

* Revert "Some logging and clear stalled blocks head"

This reverts commit 757641d9b817ae8b63fec684759b0815af9c4d0e.

* Reset stalled header if useless more than once

* Store useless headers in HashSet

* Add sync target to logging macro

* Don't disable useless peer and fix log macro

* Clear useless headers on reset and comments

* Use custom error for collecting blocks

Previously we resued BlockImportError, however only the Invalid case and
this made little sense with the QueueFull error.

* Remove blank line

* Test for reset sync after consecutive useless headers

* Don't reset after consecutive headers when chain head

* Delete commented out imports

* Return DownloadAction from collect_blocks instead of error

* Don't reset after round complete, was causing test hangs

* Add comment explaining reset after useless

* Replace HashSet with counter for useless headers

* Refactor sync reset on bad block/queue full

* Add missing target for log message

* Fix compiler errors and test after merge

* ethcore: revert ethereum tests submodule update
2018-10-09 15:31:40 +02:00
André Silva
dc14cce7a9 ethcore: fix detection of major import (#9552)
* sync: set state to idle after sync is completed

* sync: refactor sync reset
2018-10-08 12:04:38 +01:00
Jim Posen
2fc1679886 Verify block syncing responses against requests (#9670)
* sync: Validate received BlockHeaders packets against stored request.

* sync: Validate received BlockBodies and BlockReceipts.

* sync: Fix broken tests.

* sync: Unit tests for BlockDownloader::import_headers.

* sync: Unit tests for import_{bodies,receipts}.

* tests: Add missing method doc.
2018-10-03 18:35:10 +08:00