* Better logging when backfilling ancient blocks fail
Print total blocks imported, closes#10792
* `finalize()` doesn't need Engine
Pull out call to migrated_blocks() from replace_client_db()
* More logs
* Clarify that the percentage may be misleading
* Remove replace_client_db() and replace with a straight call to restore_db()
* Include the parent_hash in UnlinkedAncientBlockChain errors
* Add a new RestorationStatus varian: Finalizing (as it can take a loooong while)
Call abort_restore() when restoration fails
* Add missing cases for new variant
* typos
* Typo and derive Debug
* Do not attempt to salvage existing blocks unless they form a complete chain back to genesis
* Fix test
* Revert "Fix test"
This reverts commit f027d4b4cb7b6c23fceec528c1711886ba9cfe4e.
* Fix test again
* Update comment
* Be careful about locks
* fix test failure
* Do not defer returning an error when the chain is broken
* Review feedback
* no hex formatting for Option
* docs: Update Readme with TOC, Contributor Guide and License sections
* docs: Simplify package descriptions in Readme
* docs: Fix typos
* docs: Update Cargo package configs adding missing descriptions
* fix: Remove machine since was accidental from diff branch. Fix contribting indentation.
* fix: Fix formatting
* remove details snippet since not code and formats badly
* review-fix: Remove duplicate description
* fix dot point formatting
* fix: Add missing description to vm package
* fix: Remove duplicate Code of Conduct and Contributor guidelines
* docs: Update Contributing for consistency with other repos
* docs: Update Readme to link to existing Code of Conduct and Contributor guidelines in .github folder
* refactor: Add missing space
* fix: Fix links to be markdown format
* review-fix: Update CONTRIBUTING.md to link to Parity Ethereum Style Guide
* docs: Fix type in .github/CONTRIBUTING.md
Co-Authored-By: David <dvdplm@gmail.com>
* docs: Fix typo in .github/CONTRIBUTING.md
Co-Authored-By: David <dvdplm@gmail.com>
* docs: Rephrase parity-clib/Cargo.toml
Co-Authored-By: David <dvdplm@gmail.com>
* review-fix: Fix whole paragraph
* review-fix: Add comment to prevent direct pushes to master branch
* review-fix: Change some rules to recommendations
* Update .github/CONTRIBUTING.md
Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>
* review-fix: Replace feature reduction with breaking changes dot point
* review-fix: Rephrase what to do when reviewing PR
* review-fix: Update parity-rpc package description and module rustdocs
* docs: Add missing fullstop
* review-fix: Update rustdoc section to show all packages first as default
* review-fix: Rename Parity Util to Parity Core Libraries
* review-fix: Rename readme too for the C bindings name change
* review-fix: Remove some docs since we do not teach the reader Rust
* review-fix: Wrap Parity Ethereum specific packages in a details section
* review-fix: Separate tools that are in this repo vs those that are not
* review-fix: Add link to ethabi on crates.io
* review-fix: Remove unnecessary extra line talking about pushing to master
* review-fix: Remove useless summary of parts of the rust book documentation section
* docs: Update .github/CONTRIBUTING.md to specify two reviewers required
Co-Authored-By: David <dvdplm@gmail.com>
* docs: Update README.md with simplification
Co-Authored-By: David <dvdplm@gmail.com>
* docs: Update README.md renaming to Test Runner instead of just Run
Co-Authored-By: David <dvdplm@gmail.com>
* review-fix: Remove hardware wallet from docs
* review-fix: Remove veto PR section of contributor doc
* review-fix: Change to test generator title
* Add a way to signal shutdown to snapshotting threads
* Pass Progress to fat_rlps() so we can abort from there too.
* Checking for abort in a single spot
* Remove nightly-only weak/strong counts
* fix warning
* Fix tests
* Add dummy impl to abort snapshots
* Add another dummy impl for TestSnapshotService
* Remove debugging code
* Return error instead of the odd Ok(())
Switch to AtomicU64
* Replace error chain for network error
* Fix usages and add manual From impls
* OnDemand Error and remove remaining dependencies
* Die error_chain, die.
* DIE
* Hasta la vista, baby
* Journal for private txs added
* Tests after adding logging to private tx fixed
* Logs getter and tests added
* Time and amount limit for logs added
* RPC method for log retrieving added
* Correct path name and time validation implemented
* References for parameters added, redundant cloning reworked
* References for parameters added, redundant cloning reworked
* Work with json moved to the separate struct
* Serialization test added
* Fixed build after the merge with head
* Documentation for methods fixed, redundant field removed
* Fixed error usages
* Timestamp trait implemented for std struct
* Commented code removed
* Remove timestamp source, rework serialization test
* u64 replaced with SystemTime
* Path made mandatory for logging
* Source of monotonic time added
* into_system_time method renamed
* Initialize time source by max from current system time and max creation time from already saved logs
* Redundant conversions removed, code a little bit reworked according to review comments
* One more redundant conversion removed, rpc call simplified
Define a new `enum` where devp2p subprotocol packet ids (currently eth and par) are defined. Additionally provide functionality to query id value and protocol of a given id object.
* fix(light-rpc): Make `light_sync` generic
The motivation behind this change is to easily mock `light-sync` to make it possible to enable `rpc-integration` tests
for the light-client.
Currently the `rpc's` requires the concrete type `sync::LightSync` which makes it very hard to do so
* fix(bad merge)
* Make the specification of the protocol to which a packet_id belongs to
explicit when calling "SyncRequester::send_packet".
* Remove "SyncIO::send" and leave only "SyncIO::send_protocol"
* Adapt tests to new code.
* Strengthen tests to check if packet_id and protocol match when sending
a devp2p packet.
* Increase the number of block bodies requested during Sync.
* Increase the number of block bodies requested during Sync.
* Check if our peer is an older parity client with the bug
of not handling large requests properly
* Add a ClientVersion struct and a ClientCapabilites trait
* Make ClientVersion its own module
* Refactor and extend use of ClientVersion
* Replace strings with ClientVersion in PeerInfo
* Group further functionality in ClientCapabilities
* Move parity client version data from tuple to its own struct.
* Implement accessor methods for ParityClientData and remove them
from ClientVersion.
* Minor fixes
* Make functions specific to parity return types specific to parity.
* Test for shorter ID strings
* Fix formatting and remove unneeded dependencies.
* Roll back Cargo.lock
* Commit last Cargo.lock
* Convert from string to ClientVersion
* * When checking if peer accepts service transactions just check
if it's parity, remove version check.
* Remove dependency on semver in ethcore-sync
* Remove unnecessary String instantiation
* Rename peer_info to peer_version
* Update RPC test helpers
* Simplify From<String>
* Parse static version string only once
* Update RPC tests to new ClientVersion struct
* Document public members
* More robust parsing of ID string
* Minor changes.
* Update version in which large block bodies requests appear.
* Update ethcore/sync/src/block_sync.rs
Co-Authored-By: elferdo <elferdo@gmail.com>
* Update util/network/src/client_version.rs
Co-Authored-By: elferdo <elferdo@gmail.com>
* Update util/network/src/client_version.rs
Co-Authored-By: elferdo <elferdo@gmail.com>
* Update tests.
* Minor fixes.
* Extract accounts from ethcore.
* Fix ethcore.
* Get rid of AccountProvider in test_helpers
* Fix rest of the code.
* Re-use EngineSigner, fix tests.
* Simplify EngineSigner to always have an Address.
* Fix RPC tests.
* Add deprecation notice to RPCs.
* Feature to disable accounts.
* extract accounts in RPC
* Run with accounts in tests.
* Fix RPC compilation and tests.
* Fix compilation of the binary.
* Fix compilation of the binary.
* Fix compilation with accounts enabled.
* Fix tests.
* Update submodule.
* Remove android.
* Use derive for Default
* Don't build secretstore by default.
* Add link to issue.
* Refresh Cargo.lock.
* Fix miner tests.
* Update rpc/Cargo.toml
Co-Authored-By: tomusdrw <tomusdrw@users.noreply.github.com>
* Fix private tests.
* Patch available private contracts during private transaction
* Key acl ABI added
* Work with secret store keys moved to the separate struct
* Private tx test refactored
* Revert "Private tx test refactored"
This reverts commit 476c132d692c7a886bc7b7cd7fe47b3d7692bd63.
* Test for calling private contract from another one added
* Test fixed
* Redundant tab removed
* ACL contract processing fixed, test added
* Merge with head
* Expect replaced with closure
* fix(ManageNetwork): replace Range -> RangeIncls
Fixes `TODO: Range should be changed to RangeInclusive once stable (https://github.com/rust-lang/rust/pull/50758)`
* fix(tests)
* fix(grumbles): off-by-one error in debug_asserts
* RangeInclusive::end() is inclusive which means that if start and end is equal the `debug_assert(range.end() >
range.start()` will fail which is shouldn't
* Replace ethcore-logger with env-logger.
* Fix logger initialization in WASM tests.
* uncomment logger initialization in secret store
* Don't use ethcore-logger in whisper.
* Move ethcore-logger within parity dir.
* Uncomment rest from secret-store.
* Use `let _ =` in private_contract for consistency
* `ok()` to `let _ =` fix in service
* Use `let _ = ` for state_db
* Add private tx enabled flag into status packet
* Error log added for the case with no peers available
* Add flag only for supported protocol versions
* Work with private handler refactored
* Log target changed
* Cargo.lock updated
* fix#10125
fix service transaction version detection if --identity is enabled, change test to match how --identity actually works
* fix wrong var
* get the index of v, not /
* idx, not idx.len()
* Update ethcore/sync/src/chain/propagator.rs
Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>
* Update ethcore/sync/src/chain/propagator.rs
Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>
* change version prefix to a const
* space
Co-Authored-By: joshua-mir <43032097+joshua-mir@users.noreply.github.com>
Fix: new blocks notifications sometimes missing in pubsub RPC
Implement new struct to pass to `new_blocks()` with extra parameter - `has_more_blocks_to_import`, which was previously used to determine whether the notification should be sent. Now it's up to each implementation to decide what to do.
Updated all implementations to behave as before, except `eth_pubsub`, which will send notification even when the queue is not empty.
Update tests.
* Add `is_idle` to LightSync to check importing status
* Use SyncStateWrapper to make sure is_idle gets updates
* Update is_major_import to use verified queue size as well
* Add comment for `is_idle`
* Add Debug to `SyncStateWrapper`
* `fn get` -> `fn into_inner`
* PIP Table Cost relative to average peers instead of max peers
* Add tracing in PIP new_cost_table
* Update stat peer_count
* Use number of leeching peers for Light serve costs
* Fix test::light_params_load_share_depends_on_max_peers (wrong type)
* Remove (now) useless test
* Remove `load_share` from LightParams.Config
Prevent div. by 0
* Add LEECHER_COUNT_FACTOR
* PR Grumble: u64 to u32 for f64 casting
* Prevent u32 overflow for avg_peer_count
* Add tests for LightSync::Statistics
* Rename db_restore => client
* First step: make it compile!
* Second step: working implementation!
* Refactoring
* Fix tests
* PR Grumbles
* PR Grumbles WIP
* Migrate ancient blocks interating backward
* Early return in block migration if snapshot is aborted
* Remove RwLock getter (PR Grumble I)
* Remove dependency on `Client`: only used Traits
* Add test for recovering aborted snapshot recovery
* Add test for migrating old blocks
* Fix build
* PR Grumble I
* PR Grumble II
* PR Grumble III
* PR Grumble IV
* PR Grumble V
* PR Grumble VI
* Fix one test
* Fix test
* PR Grumble
* PR Grumbles
* PR Grumbles II
* Fix tests
* Release RwLock earlier
* Revert Cargo.lock
* Update _update ancient block_ logic: set local in `commit`
* Update typo in ethcore/src/snapshot/service.rs
Co-Authored-By: ngotchac <ngotchac@gmail.com>
* If no subchain heads then try a different peer
* Add log when useless chain head
* Restrict ChainHead useless peer to ancient blocks
* sync: replace `limit_reorg` with `block_set` condition
* Log block set in block_sync for easier debugging
* logging macros
* Match no args in sync logging macros
* Add QueueFull error
* Only allow importing headers if the first matches requested
* WIP
* Test for chain head gaps and log
* Calc distance even with 2 heads
* Revert previous commits, preparing simple fix
This reverts commit 5f38aa885b22ebb0e3a1d60120cea69f9f322628.
* Reject headers with no gaps when ChainHead
* Reset block sync download when queue full
* Simplify check for subchain heads
* Add comment to explain subchain heads filter
* Fix is_subchain_heads check and comment
* Prevent premature round completion after restart
This is a problem on mainnet where multiple stale peer requests will
force many rounds to complete quickly, forcing the retraction.
* Reset stale old blocks request after queue full
* Revert "Reject headers with no gaps when ChainHead"
This reverts commit 0eb865539e5dee37ab34f168f5fb643300de5ace.
* Add BlockSet to BlockDownloader logging
Currently it is difficult to debug this because there are two instances,
one for OldBlocks and one for NewBlocks. This adds the BlockSet to all
log messages for easy log filtering.
* Reset OldBlocks download from last enqueued
Previously when the ancient block queue was full it would restart the
download from the last imported block, so the ones still in the queue would be
redownloaded. Keeping the existing downloader instance and just
resetting it will start again from the last enqueued block.:wq
* Ignore expired Body and Receipt requests
* Log when ancient block download being restarted
* Only request old blocks from peers with >= difficulty
https://github.com/paritytech/parity-ethereum/pull/9226 might be too
permissive and causing the behaviour of the retraction soon after the
fork block. With this change the peer difficulty has to be greater than
or euqal to our syncing difficulty, so should still fix
https://github.com/paritytech/parity-ethereum/issues/9225
* Some logging and clear stalled blocks head
* Revert "Some logging and clear stalled blocks head"
This reverts commit 757641d9b817ae8b63fec684759b0815af9c4d0e.
* Reset stalled header if useless more than once
* Store useless headers in HashSet
* Add sync target to logging macro
* Don't disable useless peer and fix log macro
* Clear useless headers on reset and comments
* Use custom error for collecting blocks
Previously we resued BlockImportError, however only the Invalid case and
this made little sense with the QueueFull error.
* Remove blank line
* Test for reset sync after consecutive useless headers
* Don't reset after consecutive headers when chain head
* Delete commented out imports
* Return DownloadAction from collect_blocks instead of error
* Don't reset after round complete, was causing test hangs
* Add comment explaining reset after useless
* Replace HashSet with counter for useless headers
* Refactor sync reset on bad block/queue full
* Add missing target for log message
* Fix compiler errors and test after merge
* ethcore: revert ethereum tests submodule update
* sync: Validate received BlockHeaders packets against stored request.
* sync: Validate received BlockBodies and BlockReceipts.
* sync: Fix broken tests.
* sync: Unit tests for BlockDownloader::import_headers.
* sync: Unit tests for import_{bodies,receipts}.
* tests: Add missing method doc.
This PR is fixing deadlock for #8918
It avoids some recursive calls on light_sync by making state check optional for Informant.
The current behavior is to display the information when informant checks if block is major version.
This change a bit the informant behavior, but not on most cases.
To remember where and how this kind of deadlock are likely to happen (not seen with Parkinglot deadlock detection because it uses std condvar), I am adding a description of the deadlock.
Also, for the reviewers there may be better solution than modifying the informant.
### Thread1
- ethcore/sync/light_sync/mod.rs
A call to the light handler through any Io (having a loop of rpc query running on like client makes the dead lock way more likely).
At the end of those calls we systematically call `maintain_sync` method.
Here maintain_sync locks `state` (it is the deadlock cause), with a write purpose
`maintain_sync` -> `begin_search` with the state locked open
`begin_search` -> lightcliennt `flush_queue` method
- ethcore/light/src/client/mod.rs
`flush_queue` -> `flush` on queue (HeaderQueue aka VerificationQueue of headers)
- ethcore/src/verification/queue/mod.rs
Condition there is some unverified or verifying content
`flush` wait on a condvar until the queue is empty. The only way to unlock condvar is that worker is empty and unlock it (so thread 2 is Verification worker).
### Thread2
A verification worker at the end of a verify loop (new block).
- ethcore/src/verification/queue/mod.rs
thread loops on `verify` method.
End of loop condition is_ready -> Import the block immediately
calls `set_sync` on QueueSignal which send a BlockVerified ClientIoMessage in inner channel (IoChannel of ClientIoMessage) using `send_sync`
- util/io/src/service_mio.rs
IoChannel `send_sync` method calls all handlers with `message` method; one of the handlers is ImportBlocks IoHandler (with a single inner Client service field)
- ethcore/light/src/client/service.rs
`message` trigger inner method `import_verified`
- core/light/src/client/mod.rs
`import_verified` at the very end notify the listeners of a new_headers, one of the listeners is Informant `listener` method
- parity/informant.rs
`newHeaders` run up to call to `is_major_importing` on its target (again clinet)
- ethcore/sync/src/light_sync/mod.rs
Here `is_major_importing` tries to get state lock (read purpose only) but cannot because of previous state lock, thus deadlock
