grassrootseconomics/openethereum
8
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

New Transaction Queue implementation (#8074)

Browse Source 
* Implementation of Verifier, Scoring and Ready.

* Queue in progress.

* TransactionPool.

* Prepare for txpool release.

* Miner refactor [WiP]

* WiP reworking miner.

* Make it compile.

* Add some docs.

* Split blockchain access to a separate file.

* Work on miner API.

* Fix ethcore tests.

* Refactor miner interface for sealing/work packages.

* Implement next nonce.

* RPC compiles.

* Implement couple of missing methdods for RPC.

* Add transaction queue listeners.

* Compiles!

* Clean-up and parallelize.

* Get rid of RefCell in header.

* Revert "Get rid of RefCell in header."

This reverts commit 0f2424c9b7319a786e1565ea2a8a6d801a21b4fb.

* Override Sync requirement.

* Fix status display.

* Unify logging.

* Extract some cheap checks.

* Measurements and optimizations.

* Fix scoring bug, heap size of bug and add cache

* Disable tx queueing and parallel verification.

* Make ethcore and ethcore-miner compile again.

* Make RPC compile again.

* Bunch of txpool tests.

* Migrate transaction queue tests.

* Nonce Cap

* Nonce cap cache and tests.

* Remove stale future transactions from the queue.

* Optimize scoring and write some tests.

* Simple penalization.

* Clean up and support for different scoring algorithms.

* Add CLI parameters for the new queue.

* Remove banning queue.

* Disable debug build.

* Change per_sender limit to be 1% instead of 5%

* Avoid cloning when propagating transactions.

* Remove old todo.

* Post-review fixes.

* Fix miner options default.

* Implement back ready transactions for light client.

* Get rid of from_pending_block

* Pass rejection reason.

* Add more details to drop.

* Rollback heap size of.

* Avoid cloning hashes when propagating and include more details on rejection.

* Fix tests.

* Introduce nonces cache.

* Remove uneccessary hashes allocation.

* Lower the mem limit.

* Re-enable parallel verification.

* Add miner log. Don't check the type if not below min_gas_price.

* Add more traces, fix disabling miner.

* Fix creating pending blocks twice on AuRa authorities.

* Fix tests.

* re-use pending blocks in AuRa

* Use reseal_min_period to prevent too frequent update_sealing.

* Fix log to contain hash not sender.

* Optimize local transactions.

* Fix aura tests.

* Update locks comments.

* Get rid of unsafe Sync impl.

* Review fixes.

* Remove excessive matches.

* Fix compilation errors.

* Use new pool in private transactions.

* Fix private-tx test.

* Fix secret store tests.

* Actually use gas_floor_target

* Fix config tests.

* Fix pool tests.

* Address grumbles.
This commit is contained in:
Tomasz Drwięga
2018-04-13 17:34:27 +02:00
committed by Marek Kotewicz
parent 03b96a7c0a
commit 1cd93e4ceb
105 changed files with 5185 additions and 5784 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
ethcore/sync/src/api.rs
View File

@@ -39,6 +39,7 @@ use light::Provider;
use light::net::{self as light_net, LightProtocol, Params as LightParams, Capabilities, Handler as LightHandler, EventContext};
use network::IpFilter;
use private_tx::PrivateTxHandler;
use transaction::UnverifiedTransaction;
/// Parity sync protocol
pub const WARP_SYNC_PROTOCOL_ID: ProtocolId = *b"par";
@@ -486,9 +487,9 @@ impl ChainNotify for EthSync {
});
}
fn transactions_received(&self, hashes: Vec<H256>, peer_id: PeerId) {
fn transactions_received(&self, txs: &[UnverifiedTransaction], peer_id: PeerId) {
let mut sync = self.eth_handler.sync.write();
sync.transactions_received(hashes, peer_id);
sync.transactions_received(txs, peer_id);
}
}

44
ethcore/sync/src/chain.rs
View File

@@ -104,15 +104,16 @@ use ethcore::header::{BlockNumber, Header as BlockHeader};
use ethcore::client::{BlockChainClient, BlockStatus, BlockId, BlockChainInfo, BlockImportError, BlockQueueInfo};
use ethcore::error::*;
use ethcore::snapshot::{ManifestData, RestorationStatus};
use transaction::PendingTransaction;
use transaction::SignedTransaction;
use sync_io::SyncIo;
use super::{WarpSync, SyncConfig};
use block_sync::{BlockDownloader, BlockRequest, BlockDownloaderImportError as DownloaderImportError, DownloadAction};
use rand::Rng;
use snapshot::{Snapshot, ChunkType};
use api::{EthProtocolInfo as PeerInfoDigest, WARP_SYNC_PROTOCOL_ID};
use transactions_stats::{TransactionsStats, Stats as TransactionStats};
use private_tx::PrivateTxHandler;
use transactions_stats::{TransactionsStats, Stats as TransactionStats};
use transaction::UnverifiedTransaction;
known_heap_size!(0, PeerInfo);
@@ -478,9 +479,9 @@ impl ChainSync {
}
/// Updates transactions were received by a peer
pub fn transactions_received(&mut self, hashes: Vec<H256>, peer_id: PeerId) {
pub fn transactions_received(&mut self, txs: &[UnverifiedTransaction], peer_id: PeerId) {
if let Some(peer_info) = self.peers.get_mut(&peer_id) {
peer_info.last_sent_transactions.extend(&hashes);
peer_info.last_sent_transactions.extend(txs.iter().map(|tx| tx.hash()));
}
}
@@ -2026,8 +2027,9 @@ impl ChainSync {
return 0;
}
let (transactions, service_transactions): (Vec<_>, Vec<_>) = transactions.into_iter()
.partition(|tx| !tx.transaction.gas_price.is_zero());
let (transactions, service_transactions): (Vec<_>, Vec<_>) = transactions.iter()
.map(|tx| tx.signed())
.partition(|tx| !tx.gas_price.is_zero());
// usual transactions could be propagated to all peers
let mut affected_peers = HashSet::new();
@@ -2062,13 +2064,13 @@ impl ChainSync {
.collect()
}
fn propagate_transactions_to_peers(&mut self, io: &mut SyncIo, peers: Vec<PeerId>, transactions: Vec<PendingTransaction>) -> HashSet<PeerId> {
fn propagate_transactions_to_peers(&mut self, io: &mut SyncIo, peers: Vec<PeerId>, transactions: Vec<&SignedTransaction>) -> HashSet<PeerId> {
let all_transactions_hashes = transactions.iter()
.map(|tx| tx.transaction.hash())
.map(|tx| tx.hash())
.collect::<HashSet<H256>>();
let all_transactions_rlp = {
let mut packet = RlpStream::new_list(transactions.len());
for tx in &transactions { packet.append(&tx.transaction); }
for tx in &transactions { packet.append(&**tx); }
packet.out()
};
@@ -2112,10 +2114,10 @@ impl ChainSync {
packet.begin_unbounded_list();
let mut pushed = 0;
for tx in &transactions {
let hash = tx.transaction.hash();
let hash = tx.hash();
if to_send.contains(&hash) {
let mut transaction = RlpStream::new();
tx.transaction.rlp_append(&mut transaction);
tx.rlp_append(&mut transaction);
let appended = packet.append_raw_checked(&transaction.drain(), 1, MAX_TRANSACTION_PACKET_SIZE);
if !appended {
// Maximal packet size reached just proceed with sending
@@ -2329,7 +2331,6 @@ mod tests {
use ethcore::header::*;
use ethcore::client::{BlockChainClient, EachBlockWith, TestBlockChainClient, ChainInfo, BlockInfo};
use ethcore::miner::MinerService;
use transaction::UnverifiedTransaction;
use private_tx::NoopPrivateTxHandler;
fn get_dummy_block(order: u32, parent_hash: H256) -> Bytes {
@@ -3064,10 +3065,9 @@ mod tests {
let queue = RwLock::new(VecDeque::new());
let ss = TestSnapshotService::new();
let mut io = TestIo::new(&mut client, &ss, &queue, None);
io.chain.miner.chain_new_blocks(io.chain, &[], &[], &[], &good_blocks);
io.chain.miner.chain_new_blocks(io.chain, &[], &[], &[], &good_blocks, false);
sync.chain_new_blocks(&mut io, &[], &[], &[], &good_blocks, &[], &[]);
assert_eq!(io.chain.miner.status().transactions_in_future_queue, 0);
assert_eq!(io.chain.miner.status().transactions_in_pending_queue, 1);
assert_eq!(io.chain.miner.ready_transactions(io.chain).len(), 1);
}
// We need to update nonce status (because we say that the block has been imported)
for h in &[good_blocks[0]] {
@@ -3078,14 +3078,12 @@ mod tests {
let queue = RwLock::new(VecDeque::new());
let ss = TestSnapshotService::new();
let mut io = TestIo::new(&client, &ss, &queue, None);
io.chain.miner.chain_new_blocks(io.chain, &[], &[], &good_blocks, &retracted_blocks);
io.chain.miner.chain_new_blocks(io.chain, &[], &[], &good_blocks, &retracted_blocks, false);
sync.chain_new_blocks(&mut io, &[], &[], &good_blocks, &retracted_blocks, &[], &[]);
}
// then
let status = client.miner.status();
assert_eq!(status.transactions_in_pending_queue, 1);
assert_eq!(status.transactions_in_future_queue, 0);
assert_eq!(client.miner.ready_transactions(&client).len(), 1);
}
#[test]
@@ -3106,13 +3104,11 @@ mod tests {
// when
sync.chain_new_blocks(&mut io, &[], &[], &[], &good_blocks, &[], &[]);
assert_eq!(io.chain.miner.status().transactions_in_future_queue, 0);
assert_eq!(io.chain.miner.status().transactions_in_pending_queue, 0);
assert_eq!(io.chain.miner.queue_status().status.transaction_count, 0);
sync.chain_new_blocks(&mut io, &[], &[], &good_blocks, &retracted_blocks, &[], &[]);
// then
let status = io.chain.miner.status();
assert_eq!(status.transactions_in_pending_queue, 0);
assert_eq!(status.transactions_in_future_queue, 0);
let status = io.chain.miner.queue_status();
assert_eq!(status.status.transaction_count, 0);
}
}

38
ethcore/sync/src/tests/consensus.rs
View File

@@ -52,8 +52,8 @@ fn authority_round() {
let io_handler0: Arc<IoHandler<ClientIoMessage>> = Arc::new(TestIoHandler::new(net.peer(0).chain.clone()));
let io_handler1: Arc<IoHandler<ClientIoMessage>> = Arc::new(TestIoHandler::new(net.peer(1).chain.clone()));
// Push transaction to both clients. Only one of them gets lucky to produce a block.
net.peer(0).chain.miner().set_engine_signer(s0.address(), "".to_owned()).unwrap();
net.peer(1).chain.miner().set_engine_signer(s1.address(), "".to_owned()).unwrap();
net.peer(0).miner.set_author(s0.address(), Some("".into())).unwrap();
net.peer(1).miner.set_author(s1.address(), Some("".to_owned())).unwrap();
net.peer(0).chain.engine().register_client(Arc::downgrade(&net.peer(0).chain) as _);
net.peer(1).chain.engine().register_client(Arc::downgrade(&net.peer(1).chain) as _);
net.peer(0).chain.set_io_channel(IoChannel::to_handler(Arc::downgrade(&io_handler1)));
@@ -61,15 +61,15 @@ fn authority_round() {
// exchange statuses
net.sync();
// Trigger block proposal
net.peer(0).chain.miner().import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 0.into(), chain_id)).unwrap();
net.peer(1).chain.miner().import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 0.into(), chain_id)).unwrap();
net.peer(0).miner.import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 0.into(), chain_id)).unwrap();
net.peer(1).miner.import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 0.into(), chain_id)).unwrap();
// Sync a block
net.sync();
assert_eq!(net.peer(0).chain.chain_info().best_block_number, 1);
assert_eq!(net.peer(1).chain.chain_info().best_block_number, 1);
net.peer(0).chain.miner().import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 1.into(), chain_id)).unwrap();
net.peer(1).chain.miner().import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 1.into(), chain_id)).unwrap();
net.peer(0).miner.import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 1.into(), chain_id)).unwrap();
net.peer(1).miner.import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 1.into(), chain_id)).unwrap();
// Move to next proposer step.
net.peer(0).chain.engine().step();
net.peer(1).chain.engine().step();
@@ -78,8 +78,8 @@ fn authority_round() {
assert_eq!(net.peer(1).chain.chain_info().best_block_number, 2);
// Fork the network with equal height.
net.peer(0).chain.miner().import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 2.into(), chain_id)).unwrap();
net.peer(1).chain.miner().import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 2.into(), chain_id)).unwrap();
net.peer(0).miner.import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 2.into(), chain_id)).unwrap();
net.peer(1).miner.import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 2.into(), chain_id)).unwrap();
// Let both nodes build one block.
net.peer(0).chain.engine().step();
let early_hash = net.peer(0).chain.chain_info().best_block_hash;
@@ -101,8 +101,8 @@ fn authority_round() {
assert_eq!(ci1.best_block_hash, early_hash);
// Selfish miner
net.peer(0).chain.miner().import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 3.into(), chain_id)).unwrap();
net.peer(1).chain.miner().import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 3.into(), chain_id)).unwrap();
net.peer(0).miner.import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 3.into(), chain_id)).unwrap();
net.peer(1).miner.import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 3.into(), chain_id)).unwrap();
// Node 0 is an earlier primary.
net.peer(0).chain.engine().step();
assert_eq!(net.peer(0).chain.chain_info().best_block_number, 4);
@@ -113,7 +113,7 @@ fn authority_round() {
// Node 1 makes 2 blocks, but is a later primary on the first one.
net.peer(1).chain.engine().step();
net.peer(1).chain.engine().step();
net.peer(1).chain.miner().import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 4.into(), chain_id)).unwrap();
net.peer(1).miner.import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 4.into(), chain_id)).unwrap();
net.peer(1).chain.engine().step();
net.peer(1).chain.engine().step();
assert_eq!(net.peer(1).chain.chain_info().best_block_number, 5);
@@ -139,9 +139,9 @@ fn tendermint() {
let io_handler0: Arc<IoHandler<ClientIoMessage>> = Arc::new(TestIoHandler::new(net.peer(0).chain.clone()));
let io_handler1: Arc<IoHandler<ClientIoMessage>> = Arc::new(TestIoHandler::new(net.peer(1).chain.clone()));
// Push transaction to both clients. Only one of them issues a proposal.
net.peer(0).chain.miner().set_engine_signer(s0.address(), "".to_owned()).unwrap();
net.peer(0).miner.set_author(s0.address(), Some("".into())).unwrap();
trace!(target: "poa", "Peer 0 is {}.", s0.address());
net.peer(1).chain.miner().set_engine_signer(s1.address(), "".to_owned()).unwrap();
net.peer(1).miner.set_author(s1.address(), Some("".into())).unwrap();
trace!(target: "poa", "Peer 1 is {}.", s1.address());
net.peer(0).chain.engine().register_client(Arc::downgrade(&net.peer(0).chain) as _);
net.peer(1).chain.engine().register_client(Arc::downgrade(&net.peer(1).chain) as _);
@@ -150,7 +150,7 @@ fn tendermint() {
// Exhange statuses
net.sync();
// Propose
net.peer(0).chain.miner().import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 0.into(), chain_id)).unwrap();
net.peer(0).miner.import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 0.into(), chain_id)).unwrap();
net.sync();
// Propose timeout, synchronous for now
net.peer(0).chain.engine().step();
@@ -161,7 +161,7 @@ fn tendermint() {
assert_eq!(net.peer(0).chain.chain_info().best_block_number, 1);
assert_eq!(net.peer(1).chain.chain_info().best_block_number, 1);
net.peer(1).chain.miner().import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 0.into(), chain_id)).unwrap();
net.peer(1).miner.import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 0.into(), chain_id)).unwrap();
// Commit timeout
net.peer(0).chain.engine().step();
net.peer(1).chain.engine().step();
@@ -175,8 +175,8 @@ fn tendermint() {
assert_eq!(net.peer(0).chain.chain_info().best_block_number, 2);
assert_eq!(net.peer(1).chain.chain_info().best_block_number, 2);
net.peer(0).chain.miner().import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 1.into(), chain_id)).unwrap();
net.peer(1).chain.miner().import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 1.into(), chain_id)).unwrap();
net.peer(0).miner.import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 1.into(), chain_id)).unwrap();
net.peer(1).miner.import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 1.into(), chain_id)).unwrap();
// Peers get disconnected.
// Commit
net.peer(0).chain.engine().step();
@@ -184,8 +184,8 @@ fn tendermint() {
// Propose
net.peer(0).chain.engine().step();
net.peer(1).chain.engine().step();
net.peer(0).chain.miner().import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 2.into(), chain_id)).unwrap();
net.peer(1).chain.miner().import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 2.into(), chain_id)).unwrap();
net.peer(0).miner.import_own_transaction(&*net.peer(0).chain, new_tx(s0.secret(), 2.into(), chain_id)).unwrap();
net.peer(1).miner.import_own_transaction(&*net.peer(1).chain, new_tx(s1.secret(), 2.into(), chain_id)).unwrap();
// Send different prevotes
net.sync();
// Prevote timeout

12
ethcore/sync/src/tests/helpers.rs
View File

@@ -206,6 +206,7 @@ pub trait Peer {
pub struct EthPeer<C> where C: FlushingBlockChainClient {
pub chain: Arc<C>,
pub miner: Arc<Miner>,
pub snapshot_service: Arc<TestSnapshotService>,
pub sync: RwLock<ChainSync>,
pub queue: RwLock<VecDeque<TestPacket>>,
@@ -340,6 +341,7 @@ impl TestNet<EthPeer<TestBlockChainClient>> {
sync: RwLock::new(sync),
snapshot_service: ss,
chain: Arc::new(chain),
miner: Arc::new(Miner::new_for_tests(&Spec::new_test(), None)),
queue: RwLock::new(VecDeque::new()),
private_tx_handler,
io_queue: RwLock::new(VecDeque::new()),
@@ -382,11 +384,12 @@ impl TestNet<EthPeer<EthcoreClient>> {
pub fn add_peer_with_private_config(&mut self, config: SyncConfig, spec: Spec, accounts: Option<Arc<AccountProvider>>) {
let channel = IoChannel::disconnected();
let miner = Arc::new(Miner::new_for_tests(&spec, accounts.clone()));
let client = EthcoreClient::new(
ClientConfig::default(),
&spec,
Arc::new(::kvdb_memorydb::create(::ethcore::db::NUM_COLUMNS.unwrap_or(0))),
Arc::new(Miner::with_spec_and_accounts(&spec, accounts.clone())),
miner.clone(),
channel.clone()
).unwrap();
@@ -397,6 +400,7 @@ impl TestNet<EthPeer<EthcoreClient>> {
sync: RwLock::new(sync),
snapshot_service: ss,
chain: client,
miner,
queue: RwLock::new(VecDeque::new()),
private_tx_handler,
io_queue: RwLock::new(VecDeque::new()),
@@ -408,11 +412,12 @@ impl TestNet<EthPeer<EthcoreClient>> {
}
pub fn add_peer(&mut self, config: SyncConfig, spec: Spec, accounts: Option<Arc<AccountProvider>>) {
let miner = Arc::new(Miner::new_for_tests(&spec, accounts));
let client = EthcoreClient::new(
ClientConfig::default(),
&spec,
Arc::new(::kvdb_memorydb::create(::ethcore::db::NUM_COLUMNS.unwrap_or(0))),
Arc::new(Miner::with_spec_and_accounts(&spec, accounts)),
miner.clone(),
IoChannel::disconnected(),
).unwrap();
@@ -422,8 +427,9 @@ impl TestNet<EthPeer<EthcoreClient>> {
let peer = Arc::new(EthPeer {
sync: RwLock::new(sync),
snapshot_service: ss,
chain: client,
queue: RwLock::new(VecDeque::new()),
chain: client,
miner,
private_tx_handler,
io_queue: RwLock::new(VecDeque::new()),
new_blocks_queue: RwLock::new(VecDeque::new()),