grassrootseconomics/openethereum
8
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Make ClientIoMessage generic over the Client (#10981)

Browse Source 
* Add client-traits crate
Move the BlockInfo trait to new crate

* New crate `machine`
Contains code extracted from ethcore that defines `Machine`, `Externalities` and other execution related code.

* Use new machine and client-traits crates in ethcore

* Use new crates machine and client-traits instead of ethcore where appropriate

* Fix tests

* Don't re-export so many types from ethcore::client

* Fixing more fallout from removing re-export

* fix test

* More fallout from not re-exporting types

* Add some docs

* cleanup

* import the macro edition style

* Tweak docs

* Add missing import

* remove unused ethabi_derive imports

* Use latest ethabi-contract

* Move many traits from ethcore/client/traits to client-traits crate
Initial version of extracted Engine trait

* Move snapshot related traits to the engine crate (eew)

* Move a few snapshot related types to common_types
Cleanup Executed as exported from machine crate

* fix warning

* Gradually introduce new engine crate: snapshot

* ethcore typechecks with new engine crate

* Sort out types outside ethcore

* Add an EpochVerifier to ethash and use that in Engine.epoch_verifier()
Cleanup

* Document pub members

* Sort out tests
Sort out default impls for EpochVerifier

* Add test-helpers feature and move EngineSigner impl to the right place

* Sort out tests

* Sort out tests and refactor verification types

* Fix missing traits

* More missing traits
Fix Histogram

* Fix tests and cleanup

* cleanup

* Put back needed logger import

* Don't rexport common_types from ethcore/src/client
Don't export ethcore::client::*

* Remove files no longer used
Use types from the engine crate
Explicit exports from engine::engine

* Get rid of itertools

* Move a few more traits from ethcore to client-traits: BlockChainReset, ScheduleInfo, StateClient

* Move ProvingBlockChainClient to client-traits

* Don't re-export ForkChoice and Transition from ethcore

* Address grumbles: sort imports, remove commented out code

* Fix merge resolution error

* Extract the Clique engine to own crate

* Extract NullEngine and the block_reward module from ethcore

* Extract InstantSeal engine to own crate

* Extract remaining engines

* Extract executive_state to own crate so it can be used by engine crates

* Remove snapshot stuff from the engine crate

* Put snapshot traits back in ethcore

* cleanup

* Remove stuff from ethcore

* Don't use itertools

* itertools in aura is legit-ish

* More post-merge fixes

* Re-export less types in client

* cleanup

* Extract spec to own crate

* Put back the test-helpers from basic-authority

* Fix ethcore benchmarks

* Reduce the public api of ethcore/verification

* WIP

* Add Cargo.toml

* Fix compilation outside ethcore

* Audit uses of import_verified_blocks() and remove unneeded calls
Cleanup

* cleanup

* Remove unused imports from ethcore

* Cleanup

* remove double semi-colons

* Add missing generic param

* More missing generics

* Update ethcore/block-reward/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/basic-authority/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/ethash/Cargo.toml

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* Update ethcore/engines/clique/src/lib.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* signers is already a ref

* Add an EngineType enum to tighten up Engine.name()

* Introduce Snapshotting enum to distinguish the type of snapshots a chain uses

* Rename supports_warp to snapshot_mode

* Missing import

* Update ethcore/src/snapshot/consensus/mod.rs

Co-Authored-By: Tomasz Drwięga <tomusdrw@users.noreply.github.com>

* missing import

* Fix import

* double semi

* Fix merge problem

* cleanup

* Parametrise `ClientIoMessage` with `()` for the light client

* Add impl Tick for ()

* Address review feedback

* Move ClientIoMessage to common-types

* remove superseeded fixme

* fix merge conflict errors
This commit is contained in:
David
2019-08-28 10:09:42 +02:00
committed by GitHub
parent 974b24549b
commit cd26526868
53 changed files with 542 additions and 399 deletions
Download Patch File Download Diff File Expand all files Collapse all files

214
ethcore/src/client/client.rs
View File

@@ -14,7 +14,7 @@
// You should have received a copy of the GNU General Public License
// along with Parity Ethereum. If not, see <http://www.gnu.org/licenses/>.
use std::{cmp, ops};
use std::cmp;
use std::collections::{HashSet, BTreeMap, VecDeque};
use std::sync::atomic::{AtomicUsize, AtomicBool, Ordering as AtomicOrdering};
use std::sync::{Arc, Weak};
@@ -42,14 +42,15 @@ use client::ancient_import::AncientVerifier;
use client::{
ReopenBlock, PrepareOpenBlock, ImportSealedBlock, BroadcastProposalBlock,
Call, BlockProducer, SealedBlockImporter, ChainNotify, EngineInfo,
ClientConfig, NewBlocks, ChainRoute, ChainMessageType, bad_blocks, ClientIoMessage,
ClientConfig, NewBlocks, ChainRoute, ChainMessageType, bad_blocks,
};
use client_traits::{
BlockInfo, ScheduleInfo, StateClient, BlockChainReset,
Nonce, Balance, ChainInfo, TransactionInfo, ImportBlock,
AccountData, BlockChain as BlockChainTrait, BlockChainClient,
IoClient, BadBlocks, ProvingBlockChainClient,
StateOrBlock
IoClient, BadBlocks, ProvingBlockChainClient, SnapshotClient,
DatabaseRestore, SnapshotWriter, Tick,
StateOrBlock,
};
use engine::Engine;
use machine::{
@@ -59,7 +60,7 @@ use machine::{
};
use trie_vm_factories::{Factories, VmFactory};
use miner::{Miner, MinerService};
use snapshot::{self, io as snapshot_io, SnapshotClient};
use snapshot;
use spec::Spec;
use account_state::State;
use executive_state;
@@ -71,6 +72,8 @@ use types::{
block::PreverifiedBlock,
block_status::BlockStatus,
blockchain_info::BlockChainInfo,
client_types::ClientReport,
io_message::ClientIoMessage,
encoded,
engines::{
ForkChoice,
@@ -111,44 +114,6 @@ const MAX_ANCIENT_BLOCKS_TO_IMPORT: usize = 4;
const MAX_QUEUE_SIZE_TO_SLEEP_ON: usize = 2;
const MIN_HISTORY_SIZE: u64 = 8;
/// Report on the status of a client.
#[derive(Default, Clone, Debug, Eq, PartialEq)]
pub struct ClientReport {
/// How many blocks have been imported so far.
pub blocks_imported: usize,
/// How many transactions have been applied so far.
pub transactions_applied: usize,
/// How much gas has been processed so far.
pub gas_processed: U256,
/// Memory used by state DB
pub state_db_mem: usize,
}
impl ClientReport {
/// Alter internal reporting to reflect the additional `block` has been processed.
pub fn accrue_block(&mut self, header: &Header, transactions: usize) {
self.blocks_imported += 1;
self.transactions_applied += transactions;
self.gas_processed = self.gas_processed + *header.gas_used();
}
}
impl<'a> ops::Sub<&'a ClientReport> for ClientReport {
type Output = Self;
fn sub(mut self, other: &'a ClientReport) -> Self {
let higher_mem = cmp::max(self.state_db_mem, other.state_db_mem);
let lower_mem = cmp::min(self.state_db_mem, other.state_db_mem);
self.blocks_imported -= other.blocks_imported;
self.transactions_applied -= other.transactions_applied;
self.gas_processed = self.gas_processed - other.gas_processed;
self.state_db_mem = higher_mem - lower_mem;
self
}
}
struct SleepState {
last_activity: Option<Instant>,
last_autosleep: Option<Instant>,
@@ -171,7 +136,7 @@ struct Importer {
pub verifier: Box<dyn Verifier<Client>>,
/// Queue containing pending blocks
pub block_queue: BlockQueue,
pub block_queue: BlockQueue<Client>,
/// Handles block sealing
pub miner: Arc<Miner>,
@@ -222,7 +187,7 @@ pub struct Client {
/// Flag changed by `sleep` and `wake_up` methods. Not to be confused with `enabled`.
liveness: AtomicBool,
io_channel: RwLock<IoChannel<ClientIoMessage>>,
io_channel: RwLock<IoChannel<ClientIoMessage<Self>>>,
/// List of actors to be notified on certain chain events
notify: RwLock<Vec<Weak<dyn ChainNotify>>>,
@@ -261,7 +226,7 @@ impl Importer {
pub fn new(
config: &ClientConfig,
engine: Arc<dyn Engine>,
message_channel: IoChannel<ClientIoMessage>,
message_channel: IoChannel<ClientIoMessage<Client>>,
miner: Arc<Miner>,
) -> Result<Importer, EthcoreError> {
let block_queue = BlockQueue::new(
@@ -723,7 +688,7 @@ impl Client {
spec: &Spec,
db: Arc<dyn BlockChainDB>,
miner: Arc<Miner>,
message_channel: IoChannel<ClientIoMessage>,
message_channel: IoChannel<ClientIoMessage<Self>>,
) -> Result<Arc<Client>, EthcoreError> {
let trie_spec = match config.fat_db {
true => TrieSpec::Fat,
@@ -948,11 +913,6 @@ impl Client {
Arc::new(last_hashes)
}
/// This is triggered by a message coming from a block queue when the block is ready for insertion
pub fn import_verified_blocks(&self) -> usize {
self.importer.import_verified_blocks(self)
}
// use a state-proving closure for the given block.
fn with_proving_caller<F, T>(&self, id: BlockId, with_call: F) -> T
where F: FnOnce(&MachineCall) -> T
@@ -1037,7 +997,7 @@ impl Client {
}
/// Replace io channel. Useful for testing.
pub fn set_io_channel(&self, io_channel: IoChannel<ClientIoMessage>) {
pub fn set_io_channel(&self, io_channel: IoChannel<ClientIoMessage<Self>>) {
*self.io_channel.write() = io_channel;
}
@@ -1112,15 +1072,6 @@ impl Client {
report
}
/// Tick the client.
// TODO: manage by real events.
pub fn tick(&self, prevent_sleep: bool) {
self.check_garbage();
if !prevent_sleep {
self.check_snooze();
}
}
fn check_garbage(&self) {
self.chain.read().collect_garbage();
self.importer.block_queue.collect_garbage();
@@ -1161,64 +1112,6 @@ impl Client {
}
}
/// Take a snapshot at the given block.
/// If the ID given is "latest", this will default to 1000 blocks behind.
pub fn take_snapshot<W: snapshot_io::SnapshotWriter + Send>(
&self,
writer: W,
at: BlockId,
p: &Progress,
) -> Result<(), EthcoreError> {
if let Snapshotting::Unsupported = self.engine.snapshot_mode() {
return Err(EthcoreError::Snapshot(SnapshotError::SnapshotsUnsupported));
}
let db = self.state_db.read().journal_db().boxed_clone();
let best_block_number = self.chain_info().best_block_number;
let block_number = self.block_number(at).ok_or_else(|| SnapshotError::InvalidStartingBlock(at))?;
if db.is_prunable() && self.pruning_info().earliest_state > block_number {
return Err(SnapshotError::OldBlockPrunedDB.into());
}
let history = cmp::min(self.history, 1000);
let start_hash = match at {
BlockId::Latest => {
let start_num = match db.earliest_era() {
Some(era) => cmp::max(era, best_block_number.saturating_sub(history)),
None => best_block_number.saturating_sub(history),
};
match self.block_hash(BlockId::Number(start_num)) {
Some(h) => h,
None => return Err(SnapshotError::InvalidStartingBlock(at).into()),
}
}
_ => match self.block_hash(at) {
Some(hash) => hash,
None => return Err(SnapshotError::InvalidStartingBlock(at).into()),
},
};
let processing_threads = self.config.snapshot.processing_threads;
let chunker = snapshot::chunker(self.engine.snapshot_mode()).ok_or_else(|| SnapshotError::SnapshotsUnsupported)?;
snapshot::take_snapshot(
chunker,
&self.chain.read(),
start_hash,
db.as_hash_db(),
writer,
p,
processing_threads,
)?;
Ok(())
}
/// Ask the client what the history parameter is.
pub fn pruning_history(&self) -> u64 {
self.history
}
fn block_hash(chain: &BlockChain, id: BlockId) -> Option<H256> {
match id {
BlockId::Hash(hash) => Some(hash),
@@ -1342,7 +1235,7 @@ impl Client {
}
}
impl snapshot::DatabaseRestore for Client {
impl DatabaseRestore for Client {
/// Restart the client with a new backend
fn restore_db(&self, new_db: &str) -> Result<(), EthcoreError> {
trace!(target: "snapshot", "Replacing client database with {:?}", new_db);
@@ -1426,6 +1319,11 @@ impl BlockChainReset for Client {
Ok(())
}
/// Ask the client what the history parameter is.
fn pruning_history(&self) -> u64 {
self.history
}
}
impl Nonce for Client {
@@ -1547,6 +1445,11 @@ impl ImportBlock for Client {
Err((_, e)) => Err(e),
}
}
/// Triggered by a message from a block queue when the block is ready for insertion
fn import_verified_blocks(&self) -> usize {
self.importer.import_verified_blocks(self)
}
}
impl StateClient for Client {
@@ -2341,6 +2244,18 @@ impl IoClient for Client {
}
}
}
}
impl Tick for Client {
/// Tick the client.
// TODO: manage by real events.
fn tick(&self, prevent_sleep: bool) {
self.check_garbage();
if !prevent_sleep {
self.check_snooze();
}
}
}
impl ReopenBlock for Client {
@@ -2581,7 +2496,60 @@ impl ProvingBlockChainClient for Client {
}
}
impl SnapshotClient for Client {}
impl SnapshotClient for Client {
fn take_snapshot<W: SnapshotWriter + Send>(
&self,
writer: W,
at: BlockId,
p: &Progress,
) -> Result<(), EthcoreError> {
if let Snapshotting::Unsupported = self.engine.snapshot_mode() {
return Err(EthcoreError::Snapshot(SnapshotError::SnapshotsUnsupported));
}
let db = self.state_db.read().journal_db().boxed_clone();
let best_block_number = self.chain_info().best_block_number;
let block_number = self.block_number(at).ok_or_else(|| SnapshotError::InvalidStartingBlock(at))?;
if db.is_prunable() && self.pruning_info().earliest_state > block_number {
return Err(SnapshotError::OldBlockPrunedDB.into());
}
let history = cmp::min(self.history, 1000);
let start_hash = match at {
BlockId::Latest => {
let start_num = match db.earliest_era() {
Some(era) => cmp::max(era, best_block_number.saturating_sub(history)),
None => best_block_number.saturating_sub(history),
};
match self.block_hash(BlockId::Number(start_num)) {
Some(h) => h,
None => return Err(SnapshotError::InvalidStartingBlock(at).into()),
}
}
_ => match self.block_hash(at) {
Some(hash) => hash,
None => return Err(SnapshotError::InvalidStartingBlock(at).into()),
},
};
let processing_threads = self.config.snapshot.processing_threads;
let chunker = snapshot::chunker(self.engine.snapshot_mode()).ok_or_else(|| SnapshotError::SnapshotsUnsupported)?;
snapshot::take_snapshot(
chunker,
&self.chain.read(),
start_hash,
db.as_hash_db(),
writer,
p,
processing_threads,
)?;
Ok(())
}
}
/// Returns `LocalizedReceipt` given `LocalizedTransaction`
/// and a vector of receipts from given block up to transaction index.
@@ -2641,7 +2609,7 @@ impl IoChannelQueue {
}
}
pub fn queue<F>(&self, channel: &IoChannel<ClientIoMessage>, count: usize, fun: F) -> EthcoreResult<()> where
pub fn queue<F>(&self, channel: &IoChannel<ClientIoMessage<Client>>, count: usize, fun: F) -> EthcoreResult<()> where
F: Fn(&Client) + Send + Sync + 'static,
{
let queue_size = self.currently_queued.load(AtomicOrdering::Relaxed);

56
ethcore/src/client/io_message.rs
View File

@@ -1,56 +0,0 @@
// Copyright 2015-2019 Parity Technologies (UK) Ltd.
// This file is part of Parity Ethereum.
// Parity Ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Parity Ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Parity Ethereum. If not, see <http://www.gnu.org/licenses/>.
use std::fmt;
use bytes::Bytes;
use client::Client;
use ethereum_types::H256;
use types::snapshot::ManifestData;
/// Message type for external and internal events
#[derive(Debug)]
pub enum ClientIoMessage {
/// Best Block Hash in chain has been changed
NewChainHead,
/// A block is ready
BlockVerified,
/// Begin snapshot restoration
BeginRestoration(ManifestData),
/// Feed a state chunk to the snapshot service
FeedStateChunk(H256, Bytes),
/// Feed a block chunk to the snapshot service
FeedBlockChunk(H256, Bytes),
/// Take a snapshot for the block with given number.
TakeSnapshot(u64),
/// Execute wrapped closure
Execute(Callback),
}
impl ClientIoMessage {
/// Create new `ClientIoMessage` that executes given procedure.
pub fn execute<F: Fn(&Client) + Send + Sync + 'static>(fun: F) -> Self {
ClientIoMessage::Execute(Callback(Box::new(fun)))
}
}
/// A function to invoke in the client thread.
pub struct Callback(pub Box<dyn Fn(&Client) + Send + Sync>);
impl fmt::Debug for Callback {
fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
write!(fmt, "<callback>")
}
}

4
ethcore/src/client/mod.rs
View File

@@ -22,15 +22,13 @@ mod client;
mod config;
#[cfg(any(test, feature = "test-helpers"))]
mod evm_test_client;
mod io_message;
#[cfg(any(test, feature = "test-helpers"))]
mod test_client;
pub use self::client::{Client, ClientReport};
pub use self::client::Client;
pub use self::config::{ClientConfig, DatabaseCompactionProfile, BlockChainConfig, VMType};
#[cfg(any(test, feature = "test-helpers"))]
pub use self::evm_test_client::{EvmTestClient, EvmTestError, TransactErr, TransactSuccess};
pub use self::io_message::ClientIoMessage;
#[cfg(any(test, feature = "test-helpers"))]
pub use self::test_client::{TestBlockChainClient, EachBlockWith, TestState};
pub use self::chain_notify::{ChainNotify, NewBlocks, ChainRoute, ChainRouteType, ChainMessageType};

4
ethcore/src/client/test_client.rs
View File

@@ -592,6 +592,10 @@ impl ImportBlock for TestBlockChainClient {
}
Ok(h)
}
fn import_verified_blocks(&self) -> usize {
unimplemented!("TestClient does not implement import_verified_blocks()")
}
}
impl Call for TestBlockChainClient {

1
ethcore/src/json_tests/chain.rs
View File

@@ -100,7 +100,6 @@ pub fn json_chain_test<H: FnMut(&str, HookType)>(json_data: &[u8], start_stop_ho
if let Ok(block) = Unverified::from_rlp(b) {
let _ = client.import_block(block);
client.flush_queue();
client.import_verified_blocks();
}
}
fail_unless(client.chain_info().best_block_hash == blockchain.best_block.into());

21
ethcore/src/lib.rs
View File

@@ -69,7 +69,6 @@ extern crate ethcore_io as io;
extern crate ethcore_miner;
extern crate ethereum_types;
extern crate executive_state;
extern crate trie_vm_factories;
extern crate futures;
extern crate hash_db;
extern crate itertools;
@@ -77,10 +76,6 @@ extern crate journaldb;
extern crate keccak_hash as hash;
extern crate keccak_hasher;
extern crate kvdb;
#[cfg(any(test, feature = "test-helpers"))]
extern crate kvdb_memorydb;
extern crate len_caching_lock;
extern crate machine;
extern crate memory_cache;
extern crate num_cpus;
@@ -92,18 +87,15 @@ extern crate patricia_trie_ethereum as ethtrie;
extern crate rand;
extern crate rayon;
extern crate rlp;
extern crate parity_util_mem;
extern crate parity_util_mem as malloc_size_of;
#[cfg(any(test, feature = "test-helpers"))]
extern crate rustc_hex;
extern crate serde;
extern crate spec;
extern crate state_db;
extern crate time_utils;
extern crate trace;
extern crate trie_vm_factories;
extern crate triehash_ethereum as triehash;
extern crate unexpected;
extern crate using_queue;
extern crate verification;
extern crate vm;
#[cfg(test)]
@@ -119,8 +111,8 @@ extern crate ethash;
extern crate ethkey;
#[cfg(any(test, feature = "test-helpers"))]
extern crate ethjson;
#[cfg(any(test, feature = "tempdir"))]
extern crate tempdir;
#[cfg(any(test, feature = "test-helpers"))]
extern crate kvdb_memorydb;
#[cfg(any(test, feature = "kvdb-rocksdb"))]
extern crate kvdb_rocksdb;
#[cfg(any(test, feature = "json-tests"))]
@@ -137,8 +129,12 @@ extern crate pod;
extern crate blooms_db;
#[cfg(any(test, feature = "env_logger"))]
extern crate env_logger;
#[cfg(any(test, feature = "test-helpers"))]
extern crate rustc_hex;
#[cfg(test)]
extern crate serde_json;
#[cfg(any(test, feature = "tempdir"))]
extern crate tempdir;
#[macro_use]
extern crate ethabi_contract;
@@ -162,7 +158,6 @@ pub mod block;
pub mod client;
pub mod miner;
pub mod snapshot;
pub mod verification;
#[cfg(test)]
mod tests;

36
ethcore/src/miner/miner.rs
View File

@@ -36,36 +36,30 @@ use miner::pool_client::{PoolClient, CachedNonceClient, NonceCache};
use miner;
use parking_lot::{Mutex, RwLock};
use rayon::prelude::*;
use types::transaction::{
self,
Action,
UnverifiedTransaction,
SignedTransaction,
PendingTransaction,
};
use types::{
BlockNumber,
ids::TransactionId,
block::Block,
header::Header,
ids::BlockId,
io_message::ClientIoMessage,
engines::{Seal, SealingState},
errors::{EthcoreError as Error, ExecutionError},
receipt::RichReceipt,
transaction::{
self,
Action,
UnverifiedTransaction,
SignedTransaction,
PendingTransaction,
},
};
use using_queue::{UsingQueue, GetAction};
use block::{ClosedBlock, SealedBlock};
use client::{
BlockProducer, SealedBlockImporter, ClientIoMessage,
};
use client_traits::{
BlockChain, ChainInfo, Nonce, TransactionInfo,
};
use engine::{
Engine,
signer::EngineSigner
};
use client::{BlockProducer, SealedBlockImporter, Client};
use client_traits::{BlockChain, ChainInfo, Nonce, TransactionInfo};
use engine::{Engine, signer::EngineSigner};
use machine::executive::contract_address;
use spec::Spec;
use account_state::State;
@@ -262,7 +256,7 @@ pub struct Miner {
transaction_queue: Arc<TransactionQueue>,
engine: Arc<dyn Engine>,
accounts: Arc<dyn LocalAccounts>,
io_channel: RwLock<Option<IoChannel<ClientIoMessage>>>,
io_channel: RwLock<Option<IoChannel<ClientIoMessage<Client>>>>,
service_transaction_checker: Option<ServiceTransactionChecker>,
}
@@ -346,7 +340,7 @@ impl Miner {
}
/// Sets `IoChannel`
pub fn set_io_channel(&self, io_channel: IoChannel<ClientIoMessage>) {
pub fn set_io_channel(&self, io_channel: IoChannel<ClientIoMessage<Client>>) {
*self.io_channel.write() = Some(io_channel);
}
@@ -1426,7 +1420,7 @@ impl miner::MinerService for Miner {
let accounts = self.accounts.clone();
let service_transaction_checker = self.service_transaction_checker.clone();
let cull = move |chain: &::client::Client| {
let cull = move |chain: &Client| {
let client = PoolClient::new(
chain,
&nonce_cache,
@@ -1437,7 +1431,7 @@ impl miner::MinerService for Miner {
queue.cull(client);
};
if let Err(e) = channel.send(ClientIoMessage::execute(cull)) {
if let Err(e) = channel.send(ClientIoMessage::<Client>::execute(cull)) {
warn!(target: "miner", "Error queueing cull: {:?}", e);
}
} else {

42
ethcore/src/miner/pool_client.rs
View File

@@ -221,30 +221,30 @@ impl<'a, C: 'a> CachedNonceClient<'a, C> {
impl<'a, C: 'a> NonceClient for CachedNonceClient<'a, C> where
C: Nonce + Sync,
{
fn account_nonce(&self, address: &Address) -> U256 {
if let Some(nonce) = self.cache.nonces.read().get(address) {
return *nonce;
}
fn account_nonce(&self, address: &Address) -> U256 {
if let Some(nonce) = self.cache.nonces.read().get(address) {
return *nonce;
}
// We don't check again if cache has been populated.
// It's not THAT expensive to fetch the nonce from state.
let mut cache = self.cache.nonces.write();
let nonce = self.client.latest_nonce(address);
cache.insert(*address, nonce);
// We don't check again if cache has been populated.
// It's not THAT expensive to fetch the nonce from state.
let mut cache = self.cache.nonces.write();
let nonce = self.client.latest_nonce(address);
cache.insert(*address, nonce);
if cache.len() < self.cache.limit {
return nonce
}
if cache.len() < self.cache.limit {
return nonce
}
debug!(target: "txpool", "NonceCache: reached limit.");
trace_time!("nonce_cache:clear");
debug!(target: "txpool", "NonceCache: reached limit.");
trace_time!("nonce_cache:clear");
// Remove excessive amount of entries from the cache
let to_remove: Vec<_> = cache.keys().take(self.cache.limit / 2).cloned().collect();
for x in to_remove {
cache.remove(&x);
}
// Remove excessive amount of entries from the cache
let to_remove: Vec<_> = cache.keys().take(self.cache.limit / 2).cloned().collect();
for x in to_remove {
cache.remove(&x);
}
nonce
}
nonce
}
}

15
ethcore/src/snapshot/io.rs
View File

@@ -26,6 +26,7 @@ use std::fs::{self, File};
use std::path::{Path, PathBuf};
use bytes::Bytes;
use client_traits::SnapshotWriter;
use ethereum_types::H256;
use rlp::{RlpStream, Rlp};
use types::{
@@ -35,20 +36,6 @@ use types::{
const SNAPSHOT_VERSION: u64 = 2;
/// Something which can write snapshots.
/// Writing the same chunk multiple times will lead to implementation-defined
/// behavior, and is not advised.
pub trait SnapshotWriter {
/// Write a compressed state chunk.
fn write_state_chunk(&mut self, hash: H256, chunk: &[u8]) -> io::Result<()>;
/// Write a compressed block chunk.
fn write_block_chunk(&mut self, hash: H256, chunk: &[u8]) -> io::Result<()>;
/// Complete writing. The manifest's chunk lists must be consistent
/// with the chunks written.
fn finish(self, manifest: ManifestData) -> io::Result<()> where Self: Sized;
}
// (hash, len, offset)
#[derive(RlpEncodable, RlpDecodable)]

5
ethcore/src/snapshot/mod.rs
View File

@@ -48,7 +48,8 @@ use bloom_journal::Bloom;
use num_cpus;
use types::snapshot::ManifestData;
use self::io::SnapshotWriter;
// todo[dvdplm] put back in snapshots once it's extracted
use client_traits::SnapshotWriter;
use super::state_db::StateDB;
use account_state::Account as StateAccount;
@@ -58,7 +59,7 @@ use crossbeam_utils::thread;
use rand::{Rng, rngs::OsRng};
pub use self::consensus::*;
pub use self::service::{SnapshotClient, Service, DatabaseRestore};
pub use self::service::Service;
pub use self::traits::{SnapshotService, SnapshotComponents, Rebuilder};
pub use self::watcher::Watcher;
pub use types::basic_account::BasicAccount;

42
ethcore/src/snapshot/service.rs
View File

@@ -29,16 +29,21 @@ use super::{
SnapshotService,
Rebuilder,
MAX_CHUNK_SIZE,
io::{SnapshotReader, LooseReader, SnapshotWriter, LooseWriter},
io::{SnapshotReader, LooseReader, LooseWriter},
chunker,
};
use blockchain::{BlockChain, BlockChainDB, BlockChainDBHandler};
use client::{Client, ClientIoMessage};
use client_traits::{BlockInfo, BlockChainClient, ChainInfo};
use client::Client;
// todo[dvdplm] put SnapshotWriter back in snapshots once extracted
use client_traits::{
BlockInfo, BlockChainClient, ChainInfo,
SnapshotClient, SnapshotWriter, DatabaseRestore,
};
use engine::Engine;
use hash::keccak;
use types::{
io_message::ClientIoMessage,
errors::{EthcoreError as Error, SnapshotError, SnapshotError::UnlinkedAncientBlockChain},
ids::BlockId,
snapshot::{ManifestData, Progress, RestorationStatus},
@@ -73,12 +78,6 @@ impl Drop for Guard {
}
}
/// External database restoration handler
pub trait DatabaseRestore: Send + Sync {
/// Restart with a new backend. Takes ownership of passed database and moves it to a new location.
fn restore_db(&self, new_db: &str) -> Result<(), Error>;
}
/// State restoration manager.
struct Restoration {
manifest: ManifestData,
@@ -213,10 +212,7 @@ impl Restoration {
}
/// Type alias for client io channel.
pub type Channel = IoChannel<ClientIoMessage>;
/// Trait alias for the Client Service used
pub trait SnapshotClient: BlockChainClient + BlockInfo + DatabaseRestore {}
pub type Channel = IoChannel<ClientIoMessage<Client>>;
/// Snapshot service parameters.
pub struct ServiceParams {
@@ -234,7 +230,7 @@ pub struct ServiceParams {
/// Usually "<chain hash>/snapshot"
pub snapshot_root: PathBuf,
/// A handle for database restoration.
pub client: Arc<dyn SnapshotClient>,
pub client: Arc<Client>,
}
/// `SnapshotService` implementation.
@@ -251,7 +247,7 @@ pub struct Service {
genesis_block: Bytes,
state_chunks: AtomicUsize,
block_chunks: AtomicUsize,
client: Arc<dyn SnapshotClient>,
client: Arc<Client>,
progress: Progress,
taking_snapshot: AtomicBool,
restoring_snapshot: AtomicBool,
@@ -483,7 +479,10 @@ impl Service {
/// calling this while a restoration is in progress or vice versa
/// will lead to a race condition where the first one to finish will
/// have their produced snapshot overwritten.
pub fn take_snapshot(&self, client: &Client, num: u64) -> Result<(), Error> {
pub fn take_snapshot<C>(&self, client: &C, num: u64) -> Result<(), Error>
where
C: ChainInfo + SnapshotClient
{
if self.taking_snapshot.compare_and_swap(false, true, Ordering::SeqCst) {
info!("Skipping snapshot at #{} as another one is currently in-progress.", num);
return Ok(());
@@ -905,13 +904,16 @@ impl Drop for Service {
#[cfg(test)]
mod tests {
use client::ClientIoMessage;
use client::Client;
use io::{IoService};
use spec;
use journaldb::Algorithm;
use snapshot::SnapshotService;
use super::*;
use types::snapshot::{ManifestData, RestorationStatus};
use types::{
io_message::ClientIoMessage,
snapshot::{ManifestData, RestorationStatus}
};
use tempdir::TempDir;
use test_helpers::{generate_dummy_client_with_spec_and_data, restoration_db_handler};
@@ -919,7 +921,7 @@ mod tests {
fn sends_async_messages() {
let gas_prices = vec![1.into(), 2.into(), 3.into(), 999.into()];
let client = generate_dummy_client_with_spec_and_data(spec::new_null, 400, 5, &gas_prices);
let service = IoService::<ClientIoMessage>::start().unwrap();
let service = IoService::<ClientIoMessage<Client>>::start().unwrap();
let spec = spec::new_test();
let tempdir = TempDir::new("").unwrap();
@@ -932,7 +934,7 @@ mod tests {
pruning: Algorithm::Archive,
channel: service.channel(),
snapshot_root: dir,
client: client,
client,
};
let service = Service::new(snapshot_params).unwrap();

2
ethcore/src/snapshot/tests/helpers.rs
View File

@@ -26,7 +26,7 @@ use account_db::AccountDBMut;
use types::basic_account::BasicAccount;
use blockchain::{BlockChain, BlockChainDB};
use client::Client;
use client_traits::ChainInfo;
use client_traits::{ChainInfo, SnapshotClient};
use engine::Engine;
use snapshot::{StateRebuilder};
use snapshot::io::{SnapshotReader, PackedWriter, PackedReader};

3
ethcore/src/snapshot/tests/proof_of_work.rs
View File

@@ -26,8 +26,9 @@ use types::{
use blockchain::generator::{BlockGenerator, BlockBuilder};
use blockchain::{BlockChain, ExtrasInsert};
use client_traits::SnapshotWriter;
use snapshot::{chunk_secondary, Error as SnapshotError, SnapshotComponents};
use snapshot::io::{PackedReader, PackedWriter, SnapshotReader, SnapshotWriter};
use snapshot::io::{PackedReader, PackedWriter, SnapshotReader};
use parking_lot::Mutex;
use snappy;

9
ethcore/src/snapshot/tests/service.rs
View File

@@ -22,14 +22,14 @@ use std::sync::Arc;
use tempdir::TempDir;
use blockchain::BlockProvider;
use client::{Client, ClientConfig};
use client_traits::{BlockInfo, ImportBlock};
use client_traits::{BlockInfo, ImportBlock, SnapshotWriter};
use types::{
ids::BlockId,
snapshot::Progress,
verification::Unverified,
snapshot::{ManifestData, RestorationStatus},
};
use snapshot::io::{PackedReader, PackedWriter, SnapshotReader, SnapshotWriter};
use snapshot::io::{PackedReader, PackedWriter, SnapshotReader};
use snapshot::service::{Service, ServiceParams};
use snapshot::{chunk_state, chunk_secondary, SnapshotService};
use spec;
@@ -77,7 +77,7 @@ fn restored_is_equivalent() {
};
let service = Service::new(service_params).unwrap();
service.take_snapshot(&client, NUM_BLOCKS as u64).unwrap();
service.take_snapshot(&*client, NUM_BLOCKS as u64).unwrap();
let manifest = service.manifest().unwrap();
@@ -226,7 +226,6 @@ fn keep_ancient_blocks() {
client2.import_block(Unverified::from_rlp(block.into_inner()).unwrap()).unwrap();
}
client2.import_verified_blocks();
client2.flush_queue();
// Restore the Snapshot
@@ -304,7 +303,7 @@ fn recover_aborted_recovery() {
};
let service = Service::new(service_params).unwrap();
service.take_snapshot(&client, NUM_BLOCKS as u64).unwrap();
service.take_snapshot(&*client, NUM_BLOCKS as u64).unwrap();
let manifest = service.manifest().unwrap();
service.init_restore(manifest.clone(), true).unwrap();

3
ethcore/src/snapshot/tests/state.rs
View File

@@ -24,9 +24,10 @@ use types::{
basic_account::BasicAccount,
errors::EthcoreError as Error,
};
use client_traits::SnapshotWriter;
use snapshot::account;
use snapshot::{chunk_state, Error as SnapshotError, Progress, StateRebuilder, SNAPSHOT_SUBPARTS};
use snapshot::io::{PackedReader, PackedWriter, SnapshotReader, SnapshotWriter};
use snapshot::io::{PackedReader, PackedWriter, SnapshotReader};
use super::helpers::StateProducer;
use rand::SeedableRng;
use rand_xorshift::XorShiftRng;

11
ethcore/src/snapshot/watcher.rs
View File

@@ -17,9 +17,12 @@
//! Watcher for snapshot-related chain events.
use parking_lot::Mutex;
use client::{Client, ChainNotify, NewBlocks, ClientIoMessage};
use client::{Client, ChainNotify, NewBlocks};
use client_traits::BlockInfo;
use types::ids::BlockId;
use types::{
ids::BlockId,
io_message::ClientIoMessage,
};
use io::IoChannel;
use ethereum_types::H256;
@@ -55,7 +58,7 @@ trait Broadcast: Send + Sync {
fn take_at(&self, num: Option<u64>);
}
impl Broadcast for Mutex<IoChannel<ClientIoMessage>> {
impl Broadcast for Mutex<IoChannel<ClientIoMessage<Client>>> {
fn take_at(&self, num: Option<u64>) {
let num = match num {
Some(n) => n,
@@ -83,7 +86,7 @@ impl Watcher {
/// Create a new `Watcher` which will trigger a snapshot event
/// once every `period` blocks, but only after that block is
/// `history` blocks old.
pub fn new<F>(client: Arc<Client>, sync_status: F, channel: IoChannel<ClientIoMessage>, period: u64, history: u64) -> Self
pub fn new<F>(client: Arc<Client>, sync_status: F, channel: IoChannel<ClientIoMessage<Client>>, period: u64, history: u64) -> Self
where F: 'static + Send + Sync + Fn() -> bool
{
Watcher {

3
ethcore/src/test_helpers.rs
View File

@@ -186,7 +186,6 @@ pub fn generate_dummy_client_with_spec_and_data<F>(test_spec: F, block_number: u
db = b.drain().state.drop().1;
}
client.flush_queue();
client.import_verified_blocks();
client
}
@@ -239,7 +238,6 @@ pub fn push_block_with_transactions(client: &Arc<Client>, transactions: &[Signed
}
client.flush_queue();
client.import_verified_blocks();
}
/// Creates dummy client (not test client) with corresponding blocks
@@ -261,7 +259,6 @@ pub fn get_test_client_with_blocks(blocks: Vec<Bytes>) -> Arc<Client> {
}
}
client.flush_queue();
client.import_verified_blocks();
client
}

7
ethcore/src/tests/client.rs
View File

@@ -32,7 +32,10 @@ use types::{
};
use client::{Client, ClientConfig, PrepareOpenBlock, ImportSealedBlock};
use client_traits::{BlockInfo, BlockChainClient, BlockChainReset, ChainInfo, ImportBlock};
use client_traits::{
BlockInfo, BlockChainClient, BlockChainReset, ChainInfo,
ImportBlock, Tick,
};
use spec;
use machine::executive::{Executive, TransactOptions};
use miner::{Miner, PendingOrdering, MinerService};
@@ -55,7 +58,6 @@ fn imports_from_empty() {
Arc::new(Miner::new_for_tests(&spec, None)),
IoChannel::disconnected(),
).unwrap();
client.import_verified_blocks();
client.flush_queue();
}
@@ -102,7 +104,6 @@ fn imports_good_block() {
panic!("error importing block being good by definition");
}
client.flush_queue();
client.import_verified_blocks();
let block = client.block_header(BlockId::Number(1)).unwrap();
assert!(!block.into_inner().is_empty());

1
ethcore/src/tests/trace.rs
View File

@@ -181,7 +181,6 @@ fn can_trace_block_and_uncle_reward() {
block.drain();
client.flush_queue();
client.import_verified_blocks();
// Test0. Check overall filter
let filter = TraceFilter {

50
ethcore/src/verification/canon_verifier.rs
View File

@@ -1,50 +0,0 @@
// Copyright 2015-2019 Parity Technologies (UK) Ltd.
// This file is part of Parity Ethereum.
// Parity Ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Parity Ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Parity Ethereum. If not, see <http://www.gnu.org/licenses/>.
//! Canonical verifier.
use call_contract::CallContract;
use client_traits::BlockInfo;
use engine::Engine;
use types::{
header::Header,
errors::EthcoreError as Error,
};
use super::Verifier;
use super::verification;
/// A canonical verifier -- this does full verification.
pub struct CanonVerifier;
impl<C: BlockInfo + CallContract> Verifier<C> for CanonVerifier {
fn verify_block_family(
&self,
header: &Header,
parent: &Header,
engine: &dyn Engine,
do_full: Option<verification::FullFamilyParams<C>>,
) -> Result<(), Error> {
verification::verify_block_family(header, parent, engine, do_full)
}
fn verify_block_final(&self, expected: &Header, got: &Header) -> Result<(), Error> {
verification::verify_block_final(expected, got)
}
fn verify_block_external(&self, header: &Header, engine: &dyn Engine) -> Result<(), Error> {
engine.verify_block_external(header)
}
}

68
ethcore/src/verification/mod.rs
View File

@@ -1,68 +0,0 @@
// Copyright 2015-2019 Parity Technologies (UK) Ltd.
// This file is part of Parity Ethereum.
// Parity Ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Parity Ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Parity Ethereum. If not, see <http://www.gnu.org/licenses/>.
//! Block verification utilities.
use call_contract::CallContract;
use client_traits::BlockInfo;
mod verification;
mod verifier;
pub mod queue;
mod canon_verifier;
mod noop_verifier;
pub use self::verification::FullFamilyParams;
pub use self::verifier::Verifier;
pub use self::queue::{BlockQueue, Config as QueueConfig};
use self::verification::{
verify_block_basic,
verify_block_unordered,
verify_header_params,
};
use self::canon_verifier::CanonVerifier;
use self::noop_verifier::NoopVerifier;
/// Verifier type.
#[derive(Debug, PartialEq, Clone)]
pub enum VerifierType {
/// Verifies block normally.
Canon,
/// Verifies block normally, but skips seal verification.
CanonNoSeal,
/// Does not verify block at all.
/// Used in tests.
Noop,
}
/// Create a new verifier based on type.
pub fn new<C: BlockInfo + CallContract>(v: VerifierType) -> Box<dyn Verifier<C>> {
match v {
VerifierType::Canon | VerifierType::CanonNoSeal => Box::new(CanonVerifier),
VerifierType::Noop => Box::new(NoopVerifier),
}
}
impl VerifierType {
/// Check if seal verification is enabled for this verifier type.
pub fn verifying_seal(&self) -> bool {
match *self {
VerifierType::Canon => true,
VerifierType::Noop | VerifierType::CanonNoSeal => false,
}
}
}

50
ethcore/src/verification/noop_verifier.rs
View File

@@ -1,50 +0,0 @@
// Copyright 2015-2019 Parity Technologies (UK) Ltd.
// This file is part of Parity Ethereum.
// Parity Ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Parity Ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Parity Ethereum. If not, see <http://www.gnu.org/licenses/>.
//! No-op verifier.
use call_contract::CallContract;
use client_traits::BlockInfo;
use engine::Engine;
use types::{
header::Header,
errors::EthcoreError as Error
};
use super::{verification, Verifier};
/// A no-op verifier -- this will verify everything it's given immediately.
#[allow(dead_code)]
pub struct NoopVerifier;
impl<C: BlockInfo + CallContract> Verifier<C> for NoopVerifier {
fn verify_block_family(
&self,
_: &Header,
_t: &Header,
_: &dyn Engine,
_: Option<verification::FullFamilyParams<C>>
) -> Result<(), Error> {
Ok(())
}
fn verify_block_final(&self, _expected: &Header, _got: &Header) -> Result<(), Error> {
Ok(())
}
fn verify_block_external(&self, _header: &Header, _engine: &dyn Engine) -> Result<(), Error> {
Ok(())
}
}

186
ethcore/src/verification/queue/kind.rs
View File

@@ -1,186 +0,0 @@
// Copyright 2015-2019 Parity Technologies (UK) Ltd.
// This file is part of Parity Ethereum.
// Parity Ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Parity Ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Parity Ethereum. If not, see <http://www.gnu.org/licenses/>.
//! Definition of valid items for the verification queue.
use engine::Engine;
use parity_util_mem::MallocSizeOf;
use ethereum_types::{H256, U256};
use types::errors::EthcoreError as Error;
pub use self::blocks::Blocks;
pub use self::headers::Headers;
/// Something which can produce a hash and a parent hash.
pub trait BlockLike {
/// Get the hash of this item.
fn hash(&self) -> H256;
/// Get the hash of this item's parent.
fn parent_hash(&self) -> H256;
/// Get the difficulty of this item.
fn difficulty(&self) -> U256;
}
/// Defines transitions between stages of verification.
///
/// It starts with a fallible transformation from an "input" into the unverified item.
/// This consists of quick, simply done checks as well as extracting particular data.
///
/// Then, there is a `verify` function which performs more expensive checks and
/// produces the verified output.
///
/// For correctness, the hashes produced by each stage of the pipeline should be
/// consistent.
pub trait Kind: 'static + Sized + Send + Sync {
/// The first stage: completely unverified.
type Input: Sized + Send + BlockLike + MallocSizeOf;
/// The second stage: partially verified.
type Unverified: Sized + Send + BlockLike + MallocSizeOf;
/// The third stage: completely verified.
type Verified: Sized + Send + BlockLike + MallocSizeOf;
/// Attempt to create the `Unverified` item from the input.
fn create(input: Self::Input, engine: &dyn Engine, check_seal: bool) -> Result<Self::Unverified, (Self::Input, Error)>;
/// Attempt to verify the `Unverified` item using the given engine.
fn verify(unverified: Self::Unverified, engine: &dyn Engine, check_seal: bool) -> Result<Self::Verified, Error>;
}
/// The blocks verification module.
pub mod blocks {
use super::{Kind, BlockLike};
use engine::Engine;
use types::{
block::PreverifiedBlock,
errors::{EthcoreError as Error, BlockError},
verification::Unverified,
};
use verification::{verify_block_basic, verify_block_unordered};
use ethereum_types::{H256, U256};
/// A mode for verifying blocks.
pub struct Blocks;
impl Kind for Blocks {
type Input = Unverified;
type Unverified = Unverified;
type Verified = PreverifiedBlock;
fn create(input: Self::Input, engine: &dyn Engine, check_seal: bool) -> Result<Self::Unverified, (Self::Input, Error)> {
match verify_block_basic(&input, engine, check_seal) {
Ok(()) => Ok(input),
Err(Error::Block(BlockError::TemporarilyInvalid(oob))) => {
debug!(target: "client", "Block received too early {}: {:?}", input.hash(), oob);
Err((input, BlockError::TemporarilyInvalid(oob).into()))
},
Err(e) => {
warn!(target: "client", "Stage 1 block verification failed for {}: {:?}", input.hash(), e);
Err((input, e))
}
}
}
fn verify(un: Self::Unverified, engine: &dyn Engine, check_seal: bool) -> Result<Self::Verified, Error> {
let hash = un.hash();
match verify_block_unordered(un, engine, check_seal) {
Ok(verified) => Ok(verified),
Err(e) => {
warn!(target: "client", "Stage 2 block verification failed for {}: {:?}", hash, e);
Err(e)
}
}
}
}
impl BlockLike for Unverified {
fn hash(&self) -> H256 {
self.header.hash()
}
fn parent_hash(&self) -> H256 {
self.header.parent_hash().clone()
}
fn difficulty(&self) -> U256 {
self.header.difficulty().clone()
}
}
impl BlockLike for PreverifiedBlock {
fn hash(&self) -> H256 {
self.header.hash()
}
fn parent_hash(&self) -> H256 {
self.header.parent_hash().clone()
}
fn difficulty(&self) -> U256 {
self.header.difficulty().clone()
}
}
}
/// Verification for headers.
pub mod headers {
use super::{Kind, BlockLike};
use engine::Engine;
use types::{
header::Header,
errors::EthcoreError as Error,
};
use verification::verify_header_params;
use ethereum_types::{H256, U256};
impl BlockLike for Header {
fn hash(&self) -> H256 { self.hash() }
fn parent_hash(&self) -> H256 { self.parent_hash().clone() }
fn difficulty(&self) -> U256 { self.difficulty().clone() }
}
/// A mode for verifying headers.
pub struct Headers;
impl Kind for Headers {
type Input = Header;
type Unverified = Header;
type Verified = Header;
fn create(input: Self::Input, engine: &dyn Engine, check_seal: bool) -> Result<Self::Unverified, (Self::Input, Error)> {
match verify_header_params(&input, engine, true, check_seal) {
Ok(_) => Ok(input),
Err(err) => Err((input, err))
}
}
fn verify(unverified: Self::Unverified, engine: &dyn Engine, check_seal: bool) -> Result<Self::Verified, Error> {
match check_seal {
true => engine.verify_block_unordered(&unverified,).map(|_| unverified),
false => Ok(unverified),
}
}
}
}

946
ethcore/src/verification/queue/mod.rs
View File

@@ -1,946 +0,0 @@
// Copyright 2015-2019 Parity Technologies (UK) Ltd.
// This file is part of Parity Ethereum.
// Parity Ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Parity Ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Parity Ethereum. If not, see <http://www.gnu.org/licenses/>.
//! A queue of blocks. Sits between network or other I/O and the `BlockChain`.
//! Sorts them ready for blockchain insertion.
use std::thread::{self, JoinHandle};
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering as AtomicOrdering};
use std::sync::Arc;
use std::cmp;
use std::collections::{VecDeque, HashSet, HashMap};
use parity_util_mem::{MallocSizeOf, MallocSizeOfExt};
use ethereum_types::{H256, U256};
use parking_lot::{Condvar, Mutex, RwLock};
use io::*;
use engine::Engine;
use client::ClientIoMessage;
use len_caching_lock::LenCachingMutex;
use types::{
errors::{BlockError, EthcoreError as Error, ImportError},
verification::VerificationQueueInfo as QueueInfo,
};
use self::kind::{BlockLike, Kind};
pub mod kind;
const MIN_MEM_LIMIT: usize = 16384;
const MIN_QUEUE_LIMIT: usize = 512;
/// Type alias for block queue convenience.
pub type BlockQueue = VerificationQueue<self::kind::Blocks>;
/// Type alias for header queue convenience.
pub type HeaderQueue = VerificationQueue<self::kind::Headers>;
/// Verification queue configuration
#[derive(Debug, PartialEq, Clone)]
pub struct Config {
/// Maximum number of items to keep in unverified queue.
/// When the limit is reached, is_full returns true.
pub max_queue_size: usize,
/// Maximum heap memory to use.
/// When the limit is reached, is_full returns true.
pub max_mem_use: usize,
/// Settings for the number of verifiers and adaptation strategy.
pub verifier_settings: VerifierSettings,
}
impl Default for Config {
fn default() -> Self {
Config {
max_queue_size: 30000,
max_mem_use: 50 * 1024 * 1024,
verifier_settings: VerifierSettings::default(),
}
}
}
/// Verifier settings.
#[derive(Debug, PartialEq, Clone)]
pub struct VerifierSettings {
/// Whether to scale amount of verifiers according to load.
// Todo: replace w/ strategy enum?
pub scale_verifiers: bool,
/// Beginning amount of verifiers.
pub num_verifiers: usize,
}
impl Default for VerifierSettings {
fn default() -> Self {
VerifierSettings {
scale_verifiers: false,
num_verifiers: ::num_cpus::get(),
}
}
}
// pool states
enum State {
// all threads with id < inner value are to work.
Work(usize),
Exit,
}
/// An item which is in the process of being verified.
#[derive(MallocSizeOf)]
pub struct Verifying<K: Kind> {
hash: H256,
output: Option<K::Verified>,
}
/// Status of items in the queue.
pub enum Status {
/// Currently queued.
Queued,
/// Known to be bad.
Bad,
/// Unknown.
Unknown,
}
impl Into<::types::block_status::BlockStatus> for Status {
fn into(self) -> ::types::block_status::BlockStatus {
use ::types::block_status::BlockStatus;
match self {
Status::Queued => BlockStatus::Queued,
Status::Bad => BlockStatus::Bad,
Status::Unknown => BlockStatus::Unknown,
}
}
}
// the internal queue sizes.
struct Sizes {
unverified: AtomicUsize,
verifying: AtomicUsize,
verified: AtomicUsize,
}
/// A queue of items to be verified. Sits between network or other I/O and the `BlockChain`.
/// Keeps them in the same order as inserted, minus invalid items.
pub struct VerificationQueue<K: Kind> {
engine: Arc<dyn Engine>,
more_to_verify: Arc<Condvar>,
verification: Arc<Verification<K>>,
deleting: Arc<AtomicBool>,
ready_signal: Arc<QueueSignal>,
empty: Arc<Condvar>,
processing: RwLock<HashMap<H256, U256>>, // hash to difficulty
ticks_since_adjustment: AtomicUsize,
max_queue_size: usize,
max_mem_use: usize,
scale_verifiers: bool,
verifier_handles: Vec<JoinHandle<()>>,
state: Arc<(Mutex<State>, Condvar)>,
total_difficulty: RwLock<U256>,
}
struct QueueSignal {
deleting: Arc<AtomicBool>,
signalled: AtomicBool,
message_channel: Mutex<IoChannel<ClientIoMessage>>,
}
impl QueueSignal {
fn set_sync(&self) {
// Do not signal when we are about to close
if self.deleting.load(AtomicOrdering::Relaxed) {
return;
}
if self.signalled.compare_and_swap(false, true, AtomicOrdering::Relaxed) == false {
let channel = self.message_channel.lock().clone();
if let Err(e) = channel.send_sync(ClientIoMessage::BlockVerified) {
debug!("Error sending BlockVerified message: {:?}", e);
}
}
}
fn set_async(&self) {
// Do not signal when we are about to close
if self.deleting.load(AtomicOrdering::Relaxed) {
return;
}
if self.signalled.compare_and_swap(false, true, AtomicOrdering::Relaxed) == false {
let channel = self.message_channel.lock().clone();
if let Err(e) = channel.send(ClientIoMessage::BlockVerified) {
debug!("Error sending BlockVerified message: {:?}", e);
}
}
}
fn reset(&self) {
self.signalled.store(false, AtomicOrdering::Relaxed);
}
}
struct Verification<K: Kind> {
// All locks must be captured in the order declared here.
unverified: LenCachingMutex<VecDeque<K::Unverified>>,
verifying: LenCachingMutex<VecDeque<Verifying<K>>>,
verified: LenCachingMutex<VecDeque<K::Verified>>,
bad: Mutex<HashSet<H256>>,
sizes: Sizes,
check_seal: bool,
}
impl<K: Kind> VerificationQueue<K> {
/// Creates a new queue instance.
pub fn new(config: Config, engine: Arc<dyn Engine>, message_channel: IoChannel<ClientIoMessage>, check_seal: bool) -> Self {
let verification = Arc::new(Verification {
unverified: LenCachingMutex::new(VecDeque::new()),
verifying: LenCachingMutex::new(VecDeque::new()),
verified: LenCachingMutex::new(VecDeque::new()),
bad: Mutex::new(HashSet::new()),
sizes: Sizes {
unverified: AtomicUsize::new(0),
verifying: AtomicUsize::new(0),
verified: AtomicUsize::new(0),
},
check_seal: check_seal,
});
let more_to_verify = Arc::new(Condvar::new());
let deleting = Arc::new(AtomicBool::new(false));
let ready_signal = Arc::new(QueueSignal {
deleting: deleting.clone(),
signalled: AtomicBool::new(false),
message_channel: Mutex::new(message_channel),
});
let empty = Arc::new(Condvar::new());
let scale_verifiers = config.verifier_settings.scale_verifiers;
let max_verifiers = ::num_cpus::get();
let default_amount = cmp::max(1, cmp::min(max_verifiers, config.verifier_settings.num_verifiers));
// if `auto-scaling` is enabled spawn up extra threads as they might be needed
// otherwise just spawn the number of threads specified by the config
let number_of_threads = if scale_verifiers {
max_verifiers
} else {
cmp::min(default_amount, max_verifiers)
};
let state = Arc::new((Mutex::new(State::Work(default_amount)), Condvar::new()));
let mut verifier_handles = Vec::with_capacity(number_of_threads);
debug!(target: "verification", "Allocating {} verifiers, {} initially active", number_of_threads, default_amount);
debug!(target: "verification", "Verifier auto-scaling {}", if scale_verifiers { "enabled" } else { "disabled" });
for i in 0..number_of_threads {
debug!(target: "verification", "Adding verification thread #{}", i);
let verification = verification.clone();
let engine = engine.clone();
let wait = more_to_verify.clone();
let ready = ready_signal.clone();
let empty = empty.clone();
let state = state.clone();
let handle = thread::Builder::new()
.name(format!("Verifier #{}", i))
.spawn(move || {
VerificationQueue::verify(
verification,
engine,
wait,
ready,
empty,
state,
i,
)
})
.expect("Failed to create verifier thread.");
verifier_handles.push(handle);
}
VerificationQueue {
engine: engine,
ready_signal: ready_signal,
more_to_verify: more_to_verify,
verification: verification,
deleting: deleting,
processing: RwLock::new(HashMap::new()),
empty: empty,
ticks_since_adjustment: AtomicUsize::new(0),
max_queue_size: cmp::max(config.max_queue_size, MIN_QUEUE_LIMIT),
max_mem_use: cmp::max(config.max_mem_use, MIN_MEM_LIMIT),
scale_verifiers: scale_verifiers,
verifier_handles: verifier_handles,
state: state,
total_difficulty: RwLock::new(0.into()),
}
}
fn verify(
verification: Arc<Verification<K>>,
engine: Arc<dyn Engine>,
wait: Arc<Condvar>,
ready: Arc<QueueSignal>,
empty: Arc<Condvar>,
state: Arc<(Mutex<State>, Condvar)>,
id: usize,
) {
loop {
// check current state.
{
let mut cur_state = state.0.lock();
while let State::Work(x) = *cur_state {
// sleep until this thread is required.
if id < x { break }
debug!(target: "verification", "verifier {} sleeping", id);
state.1.wait(&mut cur_state);
debug!(target: "verification", "verifier {} waking up", id);
}
if let State::Exit = *cur_state {
debug!(target: "verification", "verifier {} exiting", id);
break;
}
}
// wait for work if empty.
{
let mut unverified = verification.unverified.lock();
if unverified.is_empty() && verification.verifying.lock().is_empty() {
empty.notify_all();
}
while unverified.is_empty() {
if let State::Exit = *state.0.lock() {
debug!(target: "verification", "verifier {} exiting", id);
return;
}
wait.wait(unverified.inner_mut());
}
if let State::Exit = *state.0.lock() {
debug!(target: "verification", "verifier {} exiting", id);
return;
}
}
// do work.
let item = {
// acquire these locks before getting the item to verify.
let mut unverified = verification.unverified.lock();
let mut verifying = verification.verifying.lock();
let item = match unverified.pop_front() {
Some(item) => item,
None => continue,
};
verification.sizes.unverified.fetch_sub(item.malloc_size_of(), AtomicOrdering::SeqCst);
verifying.push_back(Verifying { hash: item.hash(), output: None });
item
};
let hash = item.hash();
let is_ready = match K::verify(item, &*engine, verification.check_seal) {
Ok(verified) => {
let mut verifying = verification.verifying.lock();
let mut idx = None;
for (i, e) in verifying.iter_mut().enumerate() {
if e.hash == hash {
idx = Some(i);
verification.sizes.verifying.fetch_add(verified.malloc_size_of(), AtomicOrdering::SeqCst);
e.output = Some(verified);
break;
}
}
if idx == Some(0) {
// we're next!
let mut verified = verification.verified.lock();
let mut bad = verification.bad.lock();
VerificationQueue::drain_verifying(&mut verifying, &mut verified, &mut bad, &verification.sizes);
true
} else {
false
}
},
Err(_) => {
let mut verifying = verification.verifying.lock();
let mut verified = verification.verified.lock();
let mut bad = verification.bad.lock();
bad.insert(hash.clone());
verifying.retain(|e| e.hash != hash);
if verifying.front().map_or(false, |x| x.output.is_some()) {
VerificationQueue::drain_verifying(&mut verifying, &mut verified, &mut bad, &verification.sizes);
true
} else {
false
}
}
};
if is_ready {
// Import the block immediately
ready.set_sync();
}
}
}
fn drain_verifying(
verifying: &mut VecDeque<Verifying<K>>,
verified: &mut VecDeque<K::Verified>,
bad: &mut HashSet<H256>,
sizes: &Sizes,
) {
let mut removed_size = 0;
let mut inserted_size = 0;
while let Some(output) = verifying.front_mut().and_then(|x| x.output.take()) {
assert!(verifying.pop_front().is_some());
let size = output.malloc_size_of();
removed_size += size;
if bad.contains(&output.parent_hash()) {
bad.insert(output.hash());
} else {
inserted_size += size;
verified.push_back(output);
}
}
sizes.verifying.fetch_sub(removed_size, AtomicOrdering::SeqCst);
sizes.verified.fetch_add(inserted_size, AtomicOrdering::SeqCst);
}
/// Clear the queue and stop verification activity.
pub fn clear(&self) {
let mut unverified = self.verification.unverified.lock();
let mut verifying = self.verification.verifying.lock();
let mut verified = self.verification.verified.lock();
unverified.clear();
verifying.clear();
verified.clear();
let sizes = &self.verification.sizes;
sizes.unverified.store(0, AtomicOrdering::Release);
sizes.verifying.store(0, AtomicOrdering::Release);
sizes.verified.store(0, AtomicOrdering::Release);
*self.total_difficulty.write() = 0.into();
self.processing.write().clear();
}
/// Wait for unverified queue to be empty
pub fn flush(&self) {
let mut unverified = self.verification.unverified.lock();
while !unverified.is_empty() || !self.verification.verifying.lock().is_empty() {
self.empty.wait(unverified.inner_mut());
}
}
/// Check if the item is currently in the queue
pub fn status(&self, hash: &H256) -> Status {
if self.processing.read().contains_key(hash) {
return Status::Queued;
}
if self.verification.bad.lock().contains(hash) {
return Status::Bad;
}
Status::Unknown
}
/// Add a block to the queue.
pub fn import(&self, input: K::Input) -> Result<H256, (K::Input, Error)> {
let hash = input.hash();
{
if self.processing.read().contains_key(&hash) {
return Err((input, Error::Import(ImportError::AlreadyQueued).into()));
}
let mut bad = self.verification.bad.lock();
if bad.contains(&hash) {
return Err((input, Error::Import(ImportError::KnownBad).into()));
}
if bad.contains(&input.parent_hash()) {
bad.insert(hash);
return Err((input, Error::Import(ImportError::KnownBad).into()));
}
}
match K::create(input, &*self.engine, self.verification.check_seal) {
Ok(item) => {
self.verification.sizes.unverified.fetch_add(item.malloc_size_of(), AtomicOrdering::SeqCst);
self.processing.write().insert(hash, item.difficulty());
{
let mut td = self.total_difficulty.write();
*td = *td + item.difficulty();
}
self.verification.unverified.lock().push_back(item);
self.more_to_verify.notify_all();
Ok(hash)
},
Err((input, err)) => {
match err {
// Don't mark future blocks as bad.
Error::Block(BlockError::TemporarilyInvalid(_)) => {},
_ => {
self.verification.bad.lock().insert(hash);
}
}
Err((input, err))
}
}
}
/// Mark given item and all its children as bad. pauses verification
/// until complete.
pub fn mark_as_bad(&self, hashes: &[H256]) {
if hashes.is_empty() {
return;
}
let mut verified_lock = self.verification.verified.lock();
let verified = &mut *verified_lock;
let mut bad = self.verification.bad.lock();
let mut processing = self.processing.write();
bad.reserve(hashes.len());
for hash in hashes {
bad.insert(hash.clone());
if let Some(difficulty) = processing.remove(hash) {
let mut td = self.total_difficulty.write();
*td = *td - difficulty;
}
}
let mut new_verified = VecDeque::new();
let mut removed_size = 0;
for output in verified.drain(..) {
if bad.contains(&output.parent_hash()) {
removed_size += output.malloc_size_of();
bad.insert(output.hash());
if let Some(difficulty) = processing.remove(&output.hash()) {
let mut td = self.total_difficulty.write();
*td = *td - difficulty;
}
} else {
new_verified.push_back(output);
}
}
self.verification.sizes.verified.fetch_sub(removed_size, AtomicOrdering::SeqCst);
*verified = new_verified;
}
/// Mark given item as processed.
/// Returns true if the queue becomes empty.
pub fn mark_as_good(&self, hashes: &[H256]) -> bool {
if hashes.is_empty() {
return self.processing.read().is_empty();
}
let mut processing = self.processing.write();
for hash in hashes {
if let Some(difficulty) = processing.remove(hash) {
let mut td = self.total_difficulty.write();
*td = *td - difficulty;
}
}
processing.is_empty()
}
/// Removes up to `max` verified items from the queue
pub fn drain(&self, max: usize) -> Vec<K::Verified> {
let mut verified = self.verification.verified.lock();
let count = cmp::min(max, verified.len());
let result = verified.drain(..count).collect::<Vec<_>>();
let drained_size = result.iter().map(MallocSizeOfExt::malloc_size_of).fold(0, |a, c| a + c);
self.verification.sizes.verified.fetch_sub(drained_size, AtomicOrdering::SeqCst);
self.ready_signal.reset();
if !verified.is_empty() {
self.ready_signal.set_async();
}
result
}
/// Returns true if there is nothing currently in the queue.
pub fn is_empty(&self) -> bool {
let v = &self.verification;
v.unverified.load_len() == 0
&& v.verifying.load_len() == 0
&& v.verified.load_len() == 0
}
/// Get queue status.
pub fn queue_info(&self) -> QueueInfo {
use std::mem::size_of;
let (unverified_len, unverified_bytes) = {
let len = self.verification.unverified.load_len();
let size = self.verification.sizes.unverified.load(AtomicOrdering::Acquire);
(len, size + len * size_of::<K::Unverified>())
};
let (verifying_len, verifying_bytes) = {
let len = self.verification.verifying.load_len();
let size = self.verification.sizes.verifying.load(AtomicOrdering::Acquire);
(len, size + len * size_of::<Verifying<K>>())
};
let (verified_len, verified_bytes) = {
let len = self.verification.verified.load_len();
let size = self.verification.sizes.verified.load(AtomicOrdering::Acquire);
(len, size + len * size_of::<K::Verified>())
};
QueueInfo {
unverified_queue_size: unverified_len,
verifying_queue_size: verifying_len,
verified_queue_size: verified_len,
max_queue_size: self.max_queue_size,
max_mem_use: self.max_mem_use,
mem_used: unverified_bytes
+ verifying_bytes
+ verified_bytes
}
}
/// Get the total difficulty of all the blocks in the queue.
pub fn total_difficulty(&self) -> U256 {
self.total_difficulty.read().clone()
}
/// Get the current number of working verifiers.
pub fn num_verifiers(&self) -> usize {
match *self.state.0.lock() {
State::Work(x) => x,
State::Exit => panic!("state only set to exit on drop; queue live now; qed"),
}
}
/// Optimise memory footprint of the heap fields, and adjust the number of threads
/// to better suit the workload.
pub fn collect_garbage(&self) {
// number of ticks to average queue stats over
// when deciding whether to change the number of verifiers.
#[cfg(not(test))]
const READJUSTMENT_PERIOD: usize = 12;
#[cfg(test)]
const READJUSTMENT_PERIOD: usize = 1;
let (u_len, v_len) = {
let u_len = {
let mut q = self.verification.unverified.lock();
q.shrink_to_fit();
q.len()
};
self.verification.verifying.lock().shrink_to_fit();
let v_len = {
let mut q = self.verification.verified.lock();
q.shrink_to_fit();
q.len()
};
(u_len as isize, v_len as isize)
};
self.processing.write().shrink_to_fit();
if !self.scale_verifiers { return }
if self.ticks_since_adjustment.fetch_add(1, AtomicOrdering::SeqCst) + 1 >= READJUSTMENT_PERIOD {
self.ticks_since_adjustment.store(0, AtomicOrdering::SeqCst);
} else {
return;
}
let current = self.num_verifiers();
let diff = (v_len - u_len).abs();
let total = v_len + u_len;
self.scale_verifiers(
if u_len < 20 {
1
} else if diff <= total / 10 {
current
} else if v_len > u_len {
current - 1
} else {
current + 1
}
);
}
// wake up or sleep verifiers to get as close to the target as
// possible, never going over the amount of initially allocated threads
// or below 1.
fn scale_verifiers(&self, target: usize) {
let current = self.num_verifiers();
let target = cmp::min(self.verifier_handles.len(), target);
let target = cmp::max(1, target);
debug!(target: "verification", "Scaling from {} to {} verifiers", current, target);
*self.state.0.lock() = State::Work(target);
self.state.1.notify_all();
}
}
impl<K: Kind> Drop for VerificationQueue<K> {
fn drop(&mut self) {
trace!(target: "shutdown", "[VerificationQueue] Closing...");
self.clear();
self.deleting.store(true, AtomicOrdering::SeqCst);
// set exit state; should be done before `more_to_verify` notification.
*self.state.0.lock() = State::Exit;
self.state.1.notify_all();
// acquire this lock to force threads to reach the waiting point
// if they're in-between the exit check and the more_to_verify wait.
{
let _unverified = self.verification.unverified.lock();
self.more_to_verify.notify_all();
}
// wait for all verifier threads to join.
for thread in self.verifier_handles.drain(..) {
thread.join().expect("Propagating verifier thread panic on shutdown");
}
trace!(target: "shutdown", "[VerificationQueue] Closed.");
}
}
#[cfg(test)]
mod tests {
use io::*;
use super::{BlockQueue, Config, State};
use test_helpers::{get_good_dummy_block_seq, get_good_dummy_block};
use bytes::Bytes;
use types::{
errors::{EthcoreError, ImportError},
verification::Unverified,
view,
views::BlockView,
};
use spec;
// create a test block queue.
// auto_scaling enables verifier adjustment.
fn get_test_queue(auto_scale: bool) -> BlockQueue {
let spec = spec::new_test();
let engine = spec.engine;
let mut config = Config::default();
config.verifier_settings.scale_verifiers = auto_scale;
BlockQueue::new(config, engine, IoChannel::disconnected(), true)
}
fn get_test_config(num_verifiers: usize, is_auto_scale: bool) -> Config {
let mut config = Config::default();
config.verifier_settings.num_verifiers = num_verifiers;
config.verifier_settings.scale_verifiers = is_auto_scale;
config
}
fn new_unverified(bytes: Bytes) -> Unverified {
Unverified::from_rlp(bytes).expect("Should be valid rlp")
}
#[test]
fn can_be_created() {
// TODO better test
let spec = spec::new_test();
let engine = spec.engine;
let _ = BlockQueue::new(Config::default(), engine, IoChannel::disconnected(), true);
}
#[test]
fn can_import_blocks() {
let queue = get_test_queue(false);
if let Err(e) = queue.import(new_unverified(get_good_dummy_block())) {
panic!("error importing block that is valid by definition({:?})", e);
}
}
#[test]
fn returns_error_for_duplicates() {
let queue = get_test_queue(false);
if let Err(e) = queue.import(new_unverified(get_good_dummy_block())) {
panic!("error importing block that is valid by definition({:?})", e);
}
let duplicate_import = queue.import(new_unverified(get_good_dummy_block()));
match duplicate_import {
Err((_, e)) => {
match e {
EthcoreError::Import(ImportError::AlreadyQueued) => {},
_ => { panic!("must return AlreadyQueued error"); }
}
}
Ok(_) => { panic!("must produce error"); }
}
}
#[test]
fn returns_total_difficulty() {
let queue = get_test_queue(false);
let block = get_good_dummy_block();
let hash = view!(BlockView, &block).header().hash().clone();
if let Err(e) = queue.import(new_unverified(block)) {
panic!("error importing block that is valid by definition({:?})", e);
}
queue.flush();
assert_eq!(queue.total_difficulty(), 131072.into());
queue.drain(10);
assert_eq!(queue.total_difficulty(), 131072.into());
queue.mark_as_good(&[ hash ]);
assert_eq!(queue.total_difficulty(), 0.into());
}
#[test]
fn returns_ok_for_drained_duplicates() {
let queue = get_test_queue(false);
let block = get_good_dummy_block();
let hash = view!(BlockView, &block).header().hash().clone();
if let Err(e) = queue.import(new_unverified(block)) {
panic!("error importing block that is valid by definition({:?})", e);
}
queue.flush();
queue.drain(10);
queue.mark_as_good(&[ hash ]);
if let Err(e) = queue.import(new_unverified(get_good_dummy_block())) {
panic!("error importing block that has already been drained ({:?})", e);
}
}
#[test]
fn returns_empty_once_finished() {
let queue = get_test_queue(false);
queue.import(new_unverified(get_good_dummy_block()))
.expect("error importing block that is valid by definition");
queue.flush();
queue.drain(1);
assert!(queue.queue_info().is_empty());
}
#[test]
fn test_mem_limit() {
let spec = spec::new_test();
let engine = spec.engine;
let mut config = Config::default();
config.max_mem_use = super::MIN_MEM_LIMIT; // empty queue uses about 15000
let queue = BlockQueue::new(config, engine, IoChannel::disconnected(), true);
assert!(!queue.queue_info().is_full());
let mut blocks = get_good_dummy_block_seq(50);
for b in blocks.drain(..) {
queue.import(new_unverified(b)).unwrap();
}
assert!(queue.queue_info().is_full());
}
#[test]
fn scaling_limits() {
let max_verifiers = ::num_cpus::get();
let queue = get_test_queue(true);
queue.scale_verifiers(max_verifiers + 1);
assert!(queue.num_verifiers() < max_verifiers + 1);
queue.scale_verifiers(0);
assert!(queue.num_verifiers() == 1);
}
#[test]
fn readjust_verifiers() {
let queue = get_test_queue(true);
// put all the verifiers to sleep to ensure
// the test isn't timing sensitive.
*queue.state.0.lock() = State::Work(0);
for block in get_good_dummy_block_seq(5000) {
queue.import(new_unverified(block)).expect("Block good by definition; qed");
}
// almost all unverified == bump verifier count.
queue.collect_garbage();
assert_eq!(queue.num_verifiers(), 1);
queue.flush();
// nothing to verify == use minimum number of verifiers.
queue.collect_garbage();
assert_eq!(queue.num_verifiers(), 1);
}
#[test]
fn worker_threads_honor_specified_number_without_scaling() {
let spec = spec::new_test();
let engine = spec.engine;
let config = get_test_config(1, false);
let queue = BlockQueue::new(config, engine, IoChannel::disconnected(), true);
assert_eq!(queue.num_verifiers(), 1);
}
#[test]
fn worker_threads_specified_to_zero_should_set_to_one() {
let spec = spec::new_test();
let engine = spec.engine;
let config = get_test_config(0, false);
let queue = BlockQueue::new(config, engine, IoChannel::disconnected(), true);
assert_eq!(queue.num_verifiers(), 1);
}
#[test]
fn worker_threads_should_only_accept_max_number_cpus() {
let spec = spec::new_test();
let engine = spec.engine;
let config = get_test_config(10_000, false);
let queue = BlockQueue::new(config, engine, IoChannel::disconnected(), true);
let num_cpus = ::num_cpus::get();
assert_eq!(queue.num_verifiers(), num_cpus);
}
#[test]
fn worker_threads_scaling_with_specifed_num_of_workers() {
let num_cpus = ::num_cpus::get();
// only run the test with at least 2 CPUs
if num_cpus > 1 {
let spec = spec::new_test();
let engine = spec.engine;
let config = get_test_config(num_cpus - 1, true);
let queue = BlockQueue::new(config, engine, IoChannel::disconnected(), true);
queue.scale_verifiers(num_cpus);
assert_eq!(queue.num_verifiers(), num_cpus);
}
}
}

811
ethcore/src/verification/verification.rs
View File

@@ -1,811 +0,0 @@
// Copyright 2015-2019 Parity Technologies (UK) Ltd.
// This file is part of Parity Ethereum.
// Parity Ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Parity Ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Parity Ethereum. If not, see <http://www.gnu.org/licenses/>.
//! Block and transaction verification functions
//!
//! Block verification is done in 3 steps
//! 1. Quick verification upon adding to the block queue
//! 2. Signatures verification done in the queue.
//! 3. Final verification against the blockchain done before enactment.
use std::collections::HashSet;
use std::time::{Duration, SystemTime, UNIX_EPOCH};
use hash::keccak;
use rlp::Rlp;
use triehash::ordered_trie_root;
use unexpected::{Mismatch, OutOfBounds};
use blockchain::*;
use call_contract::CallContract;
use client_traits::BlockInfo;
use engine::Engine;
use types::{
BlockNumber,
header::Header,
errors::{EthcoreError as Error, BlockError},
engines::MAX_UNCLE_AGE,
block::PreverifiedBlock,
verification::Unverified,
};
use time_utils::CheckedSystemTime;
/// Phase 1 quick block verification. Only does checks that are cheap. Operates on a single block
pub fn verify_block_basic(block: &Unverified, engine: &dyn Engine, check_seal: bool) -> Result<(), Error> {
verify_header_params(&block.header, engine, true, check_seal)?;
verify_block_integrity(block)?;
if check_seal {
engine.verify_block_basic(&block.header)?;
}
for uncle in &block.uncles {
verify_header_params(uncle, engine, false, check_seal)?;
if check_seal {
engine.verify_block_basic(uncle)?;
}
}
for t in &block.transactions {
engine.verify_transaction_basic(t, &block.header)?;
}
Ok(())
}
/// Phase 2 verification. Perform costly checks such as transaction signatures and block nonce for ethash.
/// Still operates on a individual block
/// Returns a `PreverifiedBlock` structure populated with transactions
pub fn verify_block_unordered(block: Unverified, engine: &dyn Engine, check_seal: bool) -> Result<PreverifiedBlock, Error> {
let header = block.header;
if check_seal {
engine.verify_block_unordered(&header)?;
for uncle in &block.uncles {
engine.verify_block_unordered(uncle)?;
}
}
// Verify transactions.
let nonce_cap = if header.number() >= engine.params().dust_protection_transition {
Some((engine.params().nonce_cap_increment * header.number()).into())
} else {
None
};
let transactions = block.transactions
.into_iter()
.map(|t| {
let t = t.verify_unordered()?;
if let Some(max_nonce) = nonce_cap {
if t.nonce >= max_nonce {
return Err(BlockError::TooManyTransactions(t.sender()).into());
}
}
Ok(t)
})
.collect::<Result<Vec<_>, Error>>()?;
Ok(PreverifiedBlock {
header,
transactions,
uncles: block.uncles,
bytes: block.bytes,
})
}
/// Parameters for full verification of block family
pub struct FullFamilyParams<'a, C: BlockInfo + CallContract + 'a> {
/// Preverified block
pub block: &'a PreverifiedBlock,
/// Block provider to use during verification
pub block_provider: &'a dyn BlockProvider,
/// Engine client to use during verification
pub client: &'a C,
}
/// Phase 3 verification. Check block information against parent and uncles.
pub fn verify_block_family<C: BlockInfo + CallContract>(header: &Header, parent: &Header, engine: &dyn Engine, do_full: Option<FullFamilyParams<C>>) -> Result<(), Error> {
// TODO: verify timestamp
verify_parent(&header, &parent, engine)?;
engine.verify_block_family(&header, &parent)?;
let params = match do_full {
Some(x) => x,
None => return Ok(()),
};
verify_uncles(params.block, params.block_provider, engine)?;
for tx in &params.block.transactions {
// transactions are verified against the parent header since the current
// state wasn't available when the tx was created
engine.machine().verify_transaction(tx, parent, params.client)?;
}
Ok(())
}
fn verify_uncles(block: &PreverifiedBlock, bc: &dyn BlockProvider, engine: &dyn Engine) -> Result<(), Error> {
let header = &block.header;
let num_uncles = block.uncles.len();
let max_uncles = engine.maximum_uncle_count(header.number());
if num_uncles != 0 {
if num_uncles > max_uncles {
return Err(From::from(BlockError::TooManyUncles(OutOfBounds {
min: None,
max: Some(max_uncles),
found: num_uncles,
})));
}
let mut excluded = HashSet::new();
excluded.insert(header.hash());
let mut hash = header.parent_hash().clone();
excluded.insert(hash.clone());
for _ in 0..MAX_UNCLE_AGE {
match bc.block_details(&hash) {
Some(details) => {
excluded.insert(details.parent);
let b = bc.block(&hash)
.expect("parent already known to be stored; qed");
excluded.extend(b.uncle_hashes());
hash = details.parent;
}
None => break
}
}
let mut verified = HashSet::new();
for uncle in &block.uncles {
if excluded.contains(&uncle.hash()) {
return Err(From::from(BlockError::UncleInChain(uncle.hash())))
}
if verified.contains(&uncle.hash()) {
return Err(From::from(BlockError::DuplicateUncle(uncle.hash())))
}
// m_currentBlock.number() - uncle.number() m_cB.n - uP.n()
// 1 2
// 2
// 3
// 4
// 5
// 6 7
// (8 Invalid)
let depth = if header.number() > uncle.number() { header.number() - uncle.number() } else { 0 };
if depth > MAX_UNCLE_AGE as u64 {
return Err(From::from(BlockError::UncleTooOld(OutOfBounds { min: Some(header.number() - depth), max: Some(header.number() - 1), found: uncle.number() })));
}
else if depth < 1 {
return Err(From::from(BlockError::UncleIsBrother(OutOfBounds { min: Some(header.number() - depth), max: Some(header.number() - 1), found: uncle.number() })));
}
// cB
// cB.p^1 1 depth, valid uncle
// cB.p^2 ---/ 2
// cB.p^3 -----/ 3
// cB.p^4 -------/ 4
// cB.p^5 ---------/ 5
// cB.p^6 -----------/ 6
// cB.p^7 -------------/
// cB.p^8
let mut expected_uncle_parent = header.parent_hash().clone();
let uncle_parent = bc.block_header_data(&uncle.parent_hash()).ok_or_else(|| Error::from(BlockError::UnknownUncleParent(uncle.parent_hash().clone())))?;
for _ in 0..depth {
match bc.block_details(&expected_uncle_parent) {
Some(details) => {
expected_uncle_parent = details.parent;
},
None => break
}
}
if expected_uncle_parent != uncle_parent.hash() {
return Err(From::from(BlockError::UncleParentNotInChain(uncle_parent.hash())));
}
let uncle_parent = uncle_parent.decode()?;
verify_parent(&uncle, &uncle_parent, engine)?;
engine.verify_block_family(&uncle, &uncle_parent)?;
verified.insert(uncle.hash());
}
}
Ok(())
}
/// Phase 4 verification. Check block information against transaction enactment results,
pub fn verify_block_final(expected: &Header, got: &Header) -> Result<(), Error> {
if expected.state_root() != got.state_root() {
return Err(From::from(BlockError::InvalidStateRoot(Mismatch { expected: *expected.state_root(), found: *got.state_root() })))
}
if expected.gas_used() != got.gas_used() {
return Err(From::from(BlockError::InvalidGasUsed(Mismatch { expected: *expected.gas_used(), found: *got.gas_used() })))
}
if expected.log_bloom() != got.log_bloom() {
return Err(From::from(BlockError::InvalidLogBloom(Box::new(Mismatch { expected: *expected.log_bloom(), found: *got.log_bloom() }))))
}
if expected.receipts_root() != got.receipts_root() {
return Err(From::from(BlockError::InvalidReceiptsRoot(Mismatch { expected: *expected.receipts_root(), found: *got.receipts_root() })))
}
Ok(())
}
/// Check basic header parameters.
pub fn verify_header_params(header: &Header, engine: &dyn Engine, is_full: bool, check_seal: bool) -> Result<(), Error> {
if check_seal {
let expected_seal_fields = engine.seal_fields(header);
if header.seal().len() != expected_seal_fields {
return Err(From::from(BlockError::InvalidSealArity(
Mismatch { expected: expected_seal_fields, found: header.seal().len() }
)));
}
}
if header.number() >= From::from(BlockNumber::max_value()) {
return Err(From::from(BlockError::RidiculousNumber(OutOfBounds { max: Some(From::from(BlockNumber::max_value())), min: None, found: header.number() })))
}
if header.gas_used() > header.gas_limit() {
return Err(From::from(BlockError::TooMuchGasUsed(OutOfBounds { max: Some(*header.gas_limit()), min: None, found: *header.gas_used() })));
}
let min_gas_limit = engine.params().min_gas_limit;
if header.gas_limit() < &min_gas_limit {
return Err(From::from(BlockError::InvalidGasLimit(OutOfBounds { min: Some(min_gas_limit), max: None, found: *header.gas_limit() })));
}
if let Some(limit) = engine.maximum_gas_limit() {
if header.gas_limit() > &limit {
return Err(From::from(BlockError::InvalidGasLimit(OutOfBounds { min: None, max: Some(limit), found: *header.gas_limit() })));
}
}
let maximum_extra_data_size = engine.maximum_extra_data_size();
if header.number() != 0 && header.extra_data().len() > maximum_extra_data_size {
return Err(From::from(BlockError::ExtraDataOutOfBounds(OutOfBounds { min: None, max: Some(maximum_extra_data_size), found: header.extra_data().len() })));
}
if let Some(ref ext) = engine.machine().ethash_extensions() {
if header.number() >= ext.dao_hardfork_transition &&
header.number() <= ext.dao_hardfork_transition + 9 &&
header.extra_data()[..] != b"dao-hard-fork"[..] {
return Err(From::from(BlockError::ExtraDataOutOfBounds(OutOfBounds { min: None, max: None, found: 0 })));
}
}
if is_full {
const ACCEPTABLE_DRIFT: Duration = Duration::from_secs(15);
// this will resist overflow until `year 2037`
let max_time = SystemTime::now() + ACCEPTABLE_DRIFT;
let invalid_threshold = max_time + ACCEPTABLE_DRIFT * 9;
let timestamp = CheckedSystemTime::checked_add(UNIX_EPOCH, Duration::from_secs(header.timestamp()))
.ok_or(BlockError::TimestampOverflow)?;
if timestamp > invalid_threshold {
return Err(From::from(BlockError::InvalidTimestamp(OutOfBounds { max: Some(max_time), min: None, found: timestamp }.into())))
}
if timestamp > max_time {
return Err(From::from(BlockError::TemporarilyInvalid(OutOfBounds { max: Some(max_time), min: None, found: timestamp }.into())))
}
}
Ok(())
}
/// Check header parameters agains parent header.
fn verify_parent(header: &Header, parent: &Header, engine: &dyn Engine) -> Result<(), Error> {
assert!(header.parent_hash().is_zero() || &parent.hash() == header.parent_hash(),
"Parent hash should already have been verified; qed");
let gas_limit_divisor = engine.params().gas_limit_bound_divisor;
if !engine.is_timestamp_valid(header.timestamp(), parent.timestamp()) {
let now = SystemTime::now();
let min = CheckedSystemTime::checked_add(now, Duration::from_secs(parent.timestamp().saturating_add(1)))
.ok_or(BlockError::TimestampOverflow)?;
let found = CheckedSystemTime::checked_add(now, Duration::from_secs(header.timestamp()))
.ok_or(BlockError::TimestampOverflow)?;
return Err(From::from(BlockError::InvalidTimestamp(OutOfBounds { max: None, min: Some(min), found }.into())))
}
if header.number() != parent.number() + 1 {
return Err(From::from(BlockError::InvalidNumber(Mismatch { expected: parent.number() + 1, found: header.number() })));
}
if header.number() == 0 {
return Err(BlockError::RidiculousNumber(OutOfBounds { min: Some(1), max: None, found: header.number() }).into());
}
let parent_gas_limit = *parent.gas_limit();
let min_gas = parent_gas_limit - parent_gas_limit / gas_limit_divisor;
let max_gas = parent_gas_limit + parent_gas_limit / gas_limit_divisor;
if header.gas_limit() <= &min_gas || header.gas_limit() >= &max_gas {
return Err(From::from(BlockError::InvalidGasLimit(OutOfBounds { min: Some(min_gas), max: Some(max_gas), found: *header.gas_limit() })));
}
Ok(())
}
/// Verify block data against header: transactions root and uncles hash.
fn verify_block_integrity(block: &Unverified) -> Result<(), Error> {
let block_rlp = Rlp::new(&block.bytes);
let tx = block_rlp.at(1)?;
let expected_root = ordered_trie_root(tx.iter().map(|r| r.as_raw()));
if &expected_root != block.header.transactions_root() {
return Err(BlockError::InvalidTransactionsRoot(Mismatch {
expected: expected_root,
found: *block.header.transactions_root(),
}).into());
}
let expected_uncles = keccak(block_rlp.at(2)?.as_raw());
if &expected_uncles != block.header.uncles_hash(){
return Err(BlockError::InvalidUnclesHash(Mismatch {
expected: expected_uncles,
found: *block.header.uncles_hash(),
}).into());
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use std::collections::{BTreeMap, HashMap};
use std::time::{SystemTime, UNIX_EPOCH};
use ethereum_types::{H256, BloomRef, U256, Address};
use blockchain::{BlockDetails, TransactionAddress, BlockReceipts};
use bytes::Bytes;
use hash::keccak;
use engine::Engine;
use ethkey::{Random, Generator};
use spec;
use test_helpers::{create_test_block_with_data, create_test_block};
use types::{
encoded,
engines::params::CommonParams,
errors::BlockError::*,
transaction::{SignedTransaction, Transaction, UnverifiedTransaction, Action},
log_entry::{LogEntry, LocalizedLogEntry},
};
use rlp;
use triehash::ordered_trie_root;
fn check_ok(result: Result<(), Error>) {
result.unwrap_or_else(|e| panic!("Block verification failed: {:?}", e));
}
fn check_fail(result: Result<(), Error>, e: BlockError) {
match result {
Err(Error::Block(ref error)) if *error == e => (),
Err(other) => panic!("Block verification failed.\nExpected: {:?}\nGot: {:?}", e, other),
Ok(_) => panic!("Block verification failed.\nExpected: {:?}\nGot: Ok", e),
}
}
fn check_fail_timestamp(result: Result<(), Error>, temp: bool) {
let name = if temp { "TemporarilyInvalid" } else { "InvalidTimestamp" };
match result {
Err(Error::Block(BlockError::InvalidTimestamp(_))) if !temp => (),
Err(Error::Block(BlockError::TemporarilyInvalid(_))) if temp => (),
Err(other) => panic!("Block verification failed.\nExpected: {}\nGot: {:?}", name, other),
Ok(_) => panic!("Block verification failed.\nExpected: {}\nGot: Ok", name),
}
}
struct TestBlockChain {
blocks: HashMap<H256, Bytes>,
numbers: HashMap<BlockNumber, H256>,
}
impl Default for TestBlockChain {
fn default() -> Self {
TestBlockChain::new()
}
}
impl TestBlockChain {
pub fn new() -> Self {
TestBlockChain {
blocks: HashMap::new(),
numbers: HashMap::new(),
}
}
pub fn insert(&mut self, bytes: Bytes) {
let header = Unverified::from_rlp(bytes.clone()).unwrap().header;
let hash = header.hash();
self.blocks.insert(hash, bytes);
self.numbers.insert(header.number(), hash);
}
}
impl BlockProvider for TestBlockChain {
fn is_known(&self, hash: &H256) -> bool {
self.blocks.contains_key(hash)
}
fn first_block(&self) -> Option<H256> {
unimplemented!()
}
/// Get raw block data
fn block(&self, hash: &H256) -> Option<encoded::Block> {
self.blocks.get(hash).cloned().map(encoded::Block::new)
}
fn block_header_data(&self, hash: &H256) -> Option<encoded::Header> {
self.block(hash)
.map(|b| b.header_view().rlp().as_raw().to_vec())
.map(encoded::Header::new)
}
fn block_body(&self, hash: &H256) -> Option<encoded::Body> {
self.block(hash)
.map(|b| BlockChain::block_to_body(&b.into_inner()))
.map(encoded::Body::new)
}
fn best_ancient_block(&self) -> Option<H256> {
None
}
/// Get the familial details concerning a block.
fn block_details(&self, hash: &H256) -> Option<BlockDetails> {
self.blocks.get(hash).map(|bytes| {
let header = Unverified::from_rlp(bytes.to_vec()).unwrap().header;
BlockDetails {
number: header.number(),
total_difficulty: *header.difficulty(),
parent: *header.parent_hash(),
children: Vec::new(),
is_finalized: false,
}
})
}
fn transaction_address(&self, _hash: &H256) -> Option<TransactionAddress> {
unimplemented!()
}
/// Get the hash of given block's number.
fn block_hash(&self, index: BlockNumber) -> Option<H256> {
self.numbers.get(&index).cloned()
}
fn block_receipts(&self, _hash: &H256) -> Option<BlockReceipts> {
unimplemented!()
}
fn blocks_with_bloom<'a, B, I, II>(&self, _blooms: II, _from_block: BlockNumber, _to_block: BlockNumber) -> Vec<BlockNumber>
where BloomRef<'a>: From<B>, II: IntoIterator<Item = B, IntoIter = I> + Copy, I: Iterator<Item = B>, Self: Sized {
unimplemented!()
}
fn logs<F>(&self, _blocks: Vec<H256>, _matches: F, _limit: Option<usize>) -> Vec<LocalizedLogEntry>
where F: Fn(&LogEntry) -> bool, Self: Sized {
unimplemented!()
}
}
fn basic_test(bytes: &[u8], engine: &dyn Engine) -> Result<(), Error> {
let unverified = Unverified::from_rlp(bytes.to_vec())?;
verify_block_basic(&unverified, engine, true)
}
fn family_test<BC>(bytes: &[u8], engine: &dyn Engine, bc: &BC) -> Result<(), Error> where BC: BlockProvider {
let block = Unverified::from_rlp(bytes.to_vec()).unwrap();
let header = block.header;
let transactions: Vec<_> = block.transactions
.into_iter()
.map(SignedTransaction::new)
.collect::<Result<_,_>>()?;
// TODO: client is really meant to be used for state query here by machine
// additions that need access to state (tx filter in specific)
// no existing tests need access to test, so having this not function
// is fine.
let client = ::client::TestBlockChainClient::default();
let parent = bc.block_header_data(header.parent_hash())
.ok_or(BlockError::UnknownParent(*header.parent_hash()))?
.decode()?;
let block = PreverifiedBlock {
header,
transactions,
uncles: block.uncles,
bytes: bytes.to_vec(),
};
let full_params = FullFamilyParams {
block: &block,
block_provider: bc as &dyn BlockProvider,
client: &client,
};
verify_block_family(&block.header, &parent, engine, Some(full_params))
}
fn unordered_test(bytes: &[u8], engine: &dyn Engine) -> Result<(), Error> {
let un = Unverified::from_rlp(bytes.to_vec())?;
verify_block_unordered(un, engine, false)?;
Ok(())
}
#[test]
fn test_verify_block_basic_with_invalid_transactions() {
let spec = spec::new_test();
let engine = &*spec.engine;
let block = {
let mut rlp = rlp::RlpStream::new_list(3);
let mut header = Header::default();
// that's an invalid transaction list rlp
let invalid_transactions = vec![vec![0u8]];
header.set_transactions_root(ordered_trie_root(&invalid_transactions));
header.set_gas_limit(engine.params().min_gas_limit);
rlp.append(&header);
rlp.append_list::<Vec<u8>, _>(&invalid_transactions);
rlp.append_raw(&rlp::EMPTY_LIST_RLP, 1);
rlp.out()
};
assert!(basic_test(&block, engine).is_err());
}
#[test]
fn test_verify_block() {
use rlp::RlpStream;
// Test against morden
let mut good = Header::new();
let spec = spec::new_test();
let engine = &*spec.engine;
let min_gas_limit = engine.params().min_gas_limit;
good.set_gas_limit(min_gas_limit);
good.set_timestamp(40);
good.set_number(10);
let keypair = Random.generate().unwrap();
let tr1 = Transaction {
action: Action::Create,
value: U256::from(0),
data: Bytes::new(),
gas: U256::from(30_000),
gas_price: U256::from(40_000),
nonce: U256::one()
}.sign(keypair.secret(), None);
let tr2 = Transaction {
action: Action::Create,
value: U256::from(0),
data: Bytes::new(),
gas: U256::from(30_000),
gas_price: U256::from(40_000),
nonce: U256::from(2)
}.sign(keypair.secret(), None);
let tr3 = Transaction {
action: Action::Call(Address::from_low_u64_be(0x0)),
value: U256::from(0),
data: Bytes::new(),
gas: U256::from(30_000),
gas_price: U256::from(0),
nonce: U256::zero(),
}.null_sign(0);
let good_transactions = [ tr1.clone(), tr2.clone() ];
let eip86_transactions = [ tr3.clone() ];
let diff_inc = U256::from(0x40);
let mut parent6 = good.clone();
parent6.set_number(6);
let mut parent7 = good.clone();
parent7.set_number(7);
parent7.set_parent_hash(parent6.hash());
parent7.set_difficulty(parent6.difficulty().clone() + diff_inc);
parent7.set_timestamp(parent6.timestamp() + 10);
let mut parent8 = good.clone();
parent8.set_number(8);
parent8.set_parent_hash(parent7.hash());
parent8.set_difficulty(parent7.difficulty().clone() + diff_inc);
parent8.set_timestamp(parent7.timestamp() + 10);
let mut good_uncle1 = good.clone();
good_uncle1.set_number(9);
good_uncle1.set_parent_hash(parent8.hash());
good_uncle1.set_difficulty(parent8.difficulty().clone() + diff_inc);
good_uncle1.set_timestamp(parent8.timestamp() + 10);
let mut ex = good_uncle1.extra_data().to_vec();
ex.push(1u8);
good_uncle1.set_extra_data(ex);
let mut good_uncle2 = good.clone();
good_uncle2.set_number(8);
good_uncle2.set_parent_hash(parent7.hash());
good_uncle2.set_difficulty(parent7.difficulty().clone() + diff_inc);
good_uncle2.set_timestamp(parent7.timestamp() + 10);
let mut ex = good_uncle2.extra_data().to_vec();
ex.push(2u8);
good_uncle2.set_extra_data(ex);
let good_uncles = vec![ good_uncle1.clone(), good_uncle2.clone() ];
let mut uncles_rlp = RlpStream::new();
uncles_rlp.append_list(&good_uncles);
let good_uncles_hash = keccak(uncles_rlp.as_raw());
let good_transactions_root = ordered_trie_root(good_transactions.iter().map(|t| ::rlp::encode::<UnverifiedTransaction>(t)));
let eip86_transactions_root = ordered_trie_root(eip86_transactions.iter().map(|t| ::rlp::encode::<UnverifiedTransaction>(t)));
let mut parent = good.clone();
parent.set_number(9);
parent.set_timestamp(parent8.timestamp() + 10);
parent.set_parent_hash(parent8.hash());
parent.set_difficulty(parent8.difficulty().clone() + diff_inc);
good.set_parent_hash(parent.hash());
good.set_difficulty(parent.difficulty().clone() + diff_inc);
good.set_timestamp(parent.timestamp() + 10);
let mut bc = TestBlockChain::new();
bc.insert(create_test_block(&good));
bc.insert(create_test_block(&parent));
bc.insert(create_test_block(&parent6));
bc.insert(create_test_block(&parent7));
bc.insert(create_test_block(&parent8));
check_ok(basic_test(&create_test_block(&good), engine));
let mut bad_header = good.clone();
bad_header.set_transactions_root(eip86_transactions_root.clone());
bad_header.set_uncles_hash(good_uncles_hash.clone());
match basic_test(&create_test_block_with_data(&bad_header, &eip86_transactions, &good_uncles), engine) {
Err(Error::Transaction(ref e)) if e == &::ethkey::Error::InvalidSignature.into() => (),
e => panic!("Block verification failed.\nExpected: Transaction Error (Invalid Signature)\nGot: {:?}", e),
}
let mut header = good.clone();
header.set_transactions_root(good_transactions_root.clone());
header.set_uncles_hash(good_uncles_hash.clone());
check_ok(basic_test(&create_test_block_with_data(&header, &good_transactions, &good_uncles), engine));
header.set_gas_limit(min_gas_limit - 1);
check_fail(basic_test(&create_test_block(&header), engine),
InvalidGasLimit(OutOfBounds { min: Some(min_gas_limit), max: None, found: header.gas_limit().clone() }));
header = good.clone();
header.set_number(BlockNumber::max_value());
check_fail(basic_test(&create_test_block(&header), engine),
RidiculousNumber(OutOfBounds { max: Some(BlockNumber::max_value()), min: None, found: header.number() }));
header = good.clone();
let gas_used = header.gas_limit().clone() + 1;
header.set_gas_used(gas_used);
check_fail(basic_test(&create_test_block(&header), engine),
TooMuchGasUsed(OutOfBounds { max: Some(header.gas_limit().clone()), min: None, found: header.gas_used().clone() }));
header = good.clone();
let mut ex = header.extra_data().to_vec();
ex.resize(engine.maximum_extra_data_size() + 1, 0u8);
header.set_extra_data(ex);
check_fail(basic_test(&create_test_block(&header), engine),
ExtraDataOutOfBounds(OutOfBounds { max: Some(engine.maximum_extra_data_size()), min: None, found: header.extra_data().len() }));
header = good.clone();
let mut ex = header.extra_data().to_vec();
ex.resize(engine.maximum_extra_data_size() + 1, 0u8);
header.set_extra_data(ex);
check_fail(basic_test(&create_test_block(&header), engine),
ExtraDataOutOfBounds(OutOfBounds { max: Some(engine.maximum_extra_data_size()), min: None, found: header.extra_data().len() }));
header = good.clone();
header.set_uncles_hash(good_uncles_hash.clone());
check_fail(basic_test(&create_test_block_with_data(&header, &good_transactions, &good_uncles), engine),
InvalidTransactionsRoot(Mismatch { expected: good_transactions_root.clone(), found: header.transactions_root().clone() }));
header = good.clone();
header.set_transactions_root(good_transactions_root.clone());
check_fail(basic_test(&create_test_block_with_data(&header, &good_transactions, &good_uncles), engine),
InvalidUnclesHash(Mismatch { expected: good_uncles_hash.clone(), found: header.uncles_hash().clone() }));
check_ok(family_test(&create_test_block(&good), engine, &bc));
check_ok(family_test(&create_test_block_with_data(&good, &good_transactions, &good_uncles), engine, &bc));
header = good.clone();
header.set_parent_hash(H256::random());
check_fail(family_test(&create_test_block_with_data(&header, &good_transactions, &good_uncles), engine, &bc),
UnknownParent(header.parent_hash().clone()));
header = good.clone();
header.set_timestamp(10);
check_fail_timestamp(family_test(&create_test_block_with_data(&header, &good_transactions, &good_uncles), engine, &bc), false);
header = good.clone();
// will return `BlockError::TimestampOverflow` when timestamp > `i32::max_value()`
header.set_timestamp(i32::max_value() as u64);
check_fail_timestamp(basic_test(&create_test_block_with_data(&header, &good_transactions, &good_uncles), engine), false);
header = good.clone();
header.set_timestamp(SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 20);
check_fail_timestamp(basic_test(&create_test_block_with_data(&header, &good_transactions, &good_uncles), engine), true);
header = good.clone();
header.set_timestamp(SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs() + 10);
header.set_uncles_hash(good_uncles_hash.clone());
header.set_transactions_root(good_transactions_root.clone());
check_ok(basic_test(&create_test_block_with_data(&header, &good_transactions, &good_uncles), engine));
header = good.clone();
header.set_number(9);
check_fail(family_test(&create_test_block_with_data(&header, &good_transactions, &good_uncles), engine, &bc),
InvalidNumber(Mismatch { expected: parent.number() + 1, found: header.number() }));
header = good.clone();
let mut bad_uncles = good_uncles.clone();
bad_uncles.push(good_uncle1.clone());
check_fail(family_test(&create_test_block_with_data(&header, &good_transactions, &bad_uncles), engine, &bc),
TooManyUncles(OutOfBounds { max: Some(engine.maximum_uncle_count(header.number())), min: None, found: bad_uncles.len() }));
header = good.clone();
bad_uncles = vec![ good_uncle1.clone(), good_uncle1.clone() ];
check_fail(family_test(&create_test_block_with_data(&header, &good_transactions, &bad_uncles), engine, &bc),
DuplicateUncle(good_uncle1.hash()));
header = good.clone();
header.set_gas_limit(0.into());
header.set_difficulty("0000000000000000000000000000000000000000000000000000000000020000".parse::<U256>().unwrap());
match family_test(&create_test_block(&header), engine, &bc) {
Err(Error::Block(InvalidGasLimit(_))) => {},
Err(_) => { panic!("should be invalid difficulty fail"); },
_ => { panic!("Should be error, got Ok"); },
}
// TODO: some additional uncle checks
}
#[test]
fn dust_protection() {
use ethkey::{Generator, Random};
use types::transaction::{Transaction, Action};
use machine::Machine;
use null_engine::NullEngine;
let mut params = CommonParams::default();
params.dust_protection_transition = 0;
params.nonce_cap_increment = 2;
let mut header = Header::default();
header.set_number(1);
let keypair = Random.generate().unwrap();
let bad_transactions: Vec<_> = (0..3).map(|i| Transaction {
action: Action::Create,
value: U256::zero(),
data: Vec::new(),
gas: 0.into(),
gas_price: U256::zero(),
nonce: i.into(),
}.sign(keypair.secret(), None)).collect();
let good_transactions = [bad_transactions[0].clone(), bad_transactions[1].clone()];
let machine = Machine::regular(params, BTreeMap::new());
let engine = NullEngine::new(Default::default(), machine);
check_fail(unordered_test(&create_test_block_with_data(&header, &bad_transactions, &[]), &engine), TooManyTransactions(keypair.address()));
unordered_test(&create_test_block_with_data(&header, &good_transactions, &[]), &engine).unwrap();
}
}

45
ethcore/src/verification/verifier.rs
View File

@@ -1,45 +0,0 @@
// Copyright 2015-2019 Parity Technologies (UK) Ltd.
// This file is part of Parity Ethereum.
// Parity Ethereum is free software: you can redistribute it and/or modify
// it under the terms of the GNU General Public License as published by
// the Free Software Foundation, either version 3 of the License, or
// (at your option) any later version.
// Parity Ethereum is distributed in the hope that it will be useful,
// but WITHOUT ANY WARRANTY; without even the implied warranty of
// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
// GNU General Public License for more details.
// You should have received a copy of the GNU General Public License
// along with Parity Ethereum. If not, see <http://www.gnu.org/licenses/>.
//! A generic verifier trait.
use call_contract::CallContract;
use client_traits::BlockInfo;
use engine::Engine;
use types::{
header::Header,
errors::EthcoreError as Error,
};
use super::verification;
/// Should be used to verify blocks.
pub trait Verifier<C>: Send + Sync
where C: BlockInfo + CallContract
{
/// Verify a block relative to its parent and uncles.
fn verify_block_family(
&self,
header: &Header,
parent: &Header,
engine: &dyn Engine,
do_full: Option<verification::FullFamilyParams<C>>
) -> Result<(), Error>;
/// Do a final verification check for an enacted header vs its expected counterpart.
fn verify_block_final(&self, expected: &Header, got: &Header) -> Result<(), Error>;
/// Verify a block, inspecting external state.
fn verify_block_external(&self, header: &Header, engine: &dyn Engine) -> Result<(), Error>;
}