grassrootseconomics/openethereum
8
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

New Transaction Queue implementation (#8074)

Browse Source 
* Implementation of Verifier, Scoring and Ready.

* Queue in progress.

* TransactionPool.

* Prepare for txpool release.

* Miner refactor [WiP]

* WiP reworking miner.

* Make it compile.

* Add some docs.

* Split blockchain access to a separate file.

* Work on miner API.

* Fix ethcore tests.

* Refactor miner interface for sealing/work packages.

* Implement next nonce.

* RPC compiles.

* Implement couple of missing methdods for RPC.

* Add transaction queue listeners.

* Compiles!

* Clean-up and parallelize.

* Get rid of RefCell in header.

* Revert "Get rid of RefCell in header."

This reverts commit 0f2424c9b7319a786e1565ea2a8a6d801a21b4fb.

* Override Sync requirement.

* Fix status display.

* Unify logging.

* Extract some cheap checks.

* Measurements and optimizations.

* Fix scoring bug, heap size of bug and add cache

* Disable tx queueing and parallel verification.

* Make ethcore and ethcore-miner compile again.

* Make RPC compile again.

* Bunch of txpool tests.

* Migrate transaction queue tests.

* Nonce Cap

* Nonce cap cache and tests.

* Remove stale future transactions from the queue.

* Optimize scoring and write some tests.

* Simple penalization.

* Clean up and support for different scoring algorithms.

* Add CLI parameters for the new queue.

* Remove banning queue.

* Disable debug build.

* Change per_sender limit to be 1% instead of 5%

* Avoid cloning when propagating transactions.

* Remove old todo.

* Post-review fixes.

* Fix miner options default.

* Implement back ready transactions for light client.

* Get rid of from_pending_block

* Pass rejection reason.

* Add more details to drop.

* Rollback heap size of.

* Avoid cloning hashes when propagating and include more details on rejection.

* Fix tests.

* Introduce nonces cache.

* Remove uneccessary hashes allocation.

* Lower the mem limit.

* Re-enable parallel verification.

* Add miner log. Don't check the type if not below min_gas_price.

* Add more traces, fix disabling miner.

* Fix creating pending blocks twice on AuRa authorities.

* Fix tests.

* re-use pending blocks in AuRa

* Use reseal_min_period to prevent too frequent update_sealing.

* Fix log to contain hash not sender.

* Optimize local transactions.

* Fix aura tests.

* Update locks comments.

* Get rid of unsafe Sync impl.

* Review fixes.

* Remove excessive matches.

* Fix compilation errors.

* Use new pool in private transactions.

* Fix private-tx test.

* Fix secret store tests.

* Actually use gas_floor_target

* Fix config tests.

* Fix pool tests.

* Address grumbles.
This commit is contained in:
Tomasz Drwięga
2018-04-13 17:34:27 +02:00
committed by Marek Kotewicz
parent 03b96a7c0a
commit 1cd93e4ceb
105 changed files with 5185 additions and 5784 deletions
Download Patch File Download Diff File Expand all files Collapse all files

97
ethcore/private-tx/src/lib.rs
View File

@@ -78,12 +78,10 @@ use ethcore::executed::{Executed};
use transaction::{SignedTransaction, Transaction, Action, UnverifiedTransaction};
use ethcore::{contract_address as ethcore_contract_address};
use ethcore::client::{
Client, ChainNotify, ChainMessageType, ClientIoMessage, BlockId,
MiningBlockChainClient, ChainInfo, Nonce, CallContract
Client, ChainNotify, ChainMessageType, ClientIoMessage, BlockId, CallContract
};
use ethcore::account_provider::AccountProvider;
use ethcore_miner::transaction_queue::{TransactionDetailsProvider as TransactionQueueDetailsProvider, AccountDetails};
use ethcore::miner::MinerService;
use ethcore::miner::{self, Miner, MinerService};
use ethcore::trace::{Tracer, VMTracer};
use rustc_hex::FromHex;
@@ -95,35 +93,6 @@ use_contract!(private, "PrivateContract", "res/private.json");
/// Initialization vector length.
const INIT_VEC_LEN: usize = 16;
struct TransactionDetailsProvider<'a> {
client: &'a MiningBlockChainClient,
}
impl<'a> TransactionDetailsProvider<'a> {
pub fn new(client: &'a MiningBlockChainClient) -> Self {
TransactionDetailsProvider {
client: client,
}
}
}
impl<'a> TransactionQueueDetailsProvider for TransactionDetailsProvider<'a> {
fn fetch_account(&self, address: &Address) -> AccountDetails {
AccountDetails {
nonce: self.client.latest_nonce(address),
balance: self.client.latest_balance(address),
}
}
fn estimate_gas_required(&self, tx: &SignedTransaction) -> U256 {
tx.gas_required(&self.client.latest_schedule()).into()
}
fn is_service_transaction_acceptable(&self, _tx: &SignedTransaction) -> Result<bool, String> {
Ok(false)
}
}
/// Configurtion for private transaction provider
#[derive(Default, PartialEq, Debug, Clone)]
pub struct ProviderConfig {
@@ -154,8 +123,10 @@ pub struct Provider {
passwords: Vec<String>,
notify: RwLock<Vec<Weak<ChainNotify>>>,
transactions_for_signing: Mutex<SigningStore>,
// TODO [ToDr] Move the Mutex/RwLock inside `VerificationStore` after refactored to `drain`.
transactions_for_verification: Mutex<VerificationStore>,
client: Arc<Client>,
miner: Arc<Miner>,
accounts: Arc<AccountProvider>,
channel: IoChannel<ClientIoMessage>,
}
@@ -172,6 +143,7 @@ impl Provider where {
/// Create a new provider.
pub fn new(
client: Arc<Client>,
miner: Arc<Miner>,
accounts: Arc<AccountProvider>,
encryptor: Box<Encryptor>,
config: ProviderConfig,
@@ -186,6 +158,7 @@ impl Provider where {
transactions_for_signing: Mutex::default(),
transactions_for_verification: Mutex::default(),
client,
miner,
accounts,
channel,
})
@@ -282,6 +255,9 @@ impl Provider where {
match validation_account {
None => {
// TODO [ToDr] This still seems a bit invalid, imho we should still import the transaction to the pool.
// Importing to pool verifies correctness and nonce; here we are just blindly forwarding.
//
// Not for verification, broadcast further to peers
self.broadcast_private_transaction(rlp.into());
return Ok(());
@@ -291,29 +267,59 @@ impl Provider where {
trace!("Private transaction taken for verification");
let original_tx = self.extract_original_transaction(private_tx, &contract)?;
trace!("Validating transaction: {:?}", original_tx);
let details_provider = TransactionDetailsProvider::new(&*self.client as &MiningBlockChainClient);
let insertion_time = self.client.chain_info().best_block_number;
// Verify with the first account available
trace!("The following account will be used for verification: {:?}", validation_account);
self.transactions_for_verification.lock()
.add_transaction(original_tx, contract, validation_account, hash, &details_provider, insertion_time)?;
let nonce_cache = Default::default();
self.transactions_for_verification.lock().add_transaction(
original_tx,
contract,
validation_account,
hash,
self.pool_client(&nonce_cache),
)?;
// NOTE This will just fire `on_private_transaction_queued` but from a client thread.
// It seems that a lot of heavy work (verification) is done in this thread anyway
// it might actually make sense to decouple it from clientService and just use dedicated thread
// for both verification and execution.
self.channel.send(ClientIoMessage::NewPrivateTransaction).map_err(|_| ErrorKind::ClientIsMalformed.into())
}
}
}
fn pool_client<'a>(&'a self, nonce_cache: &'a RwLock<HashMap<Address, U256>>) -> miner::pool_client::PoolClient<'a, Client> {
let engine = self.client.engine();
let refuse_service_transactions = true;
miner::pool_client::PoolClient::new(
&*self.client,
nonce_cache,
engine,
Some(&*self.accounts),
refuse_service_transactions,
)
}
/// Private transaction for validation added into queue
pub fn on_private_transaction_queued(&self) -> Result<(), Error> {
self.process_queue()
}
/// Retrieve and verify the first available private transaction for every sender
///
/// TODO [ToDr] It seems that:
/// 1. This method will fail on any error without removing invalid transaction.
/// 2. It means that the transaction will be stuck there forever and we will never be able to make any progress.
///
/// It might be more sensible to `drain()` transactions from the queue instead and process all of them,
/// possibly printing some errors in case of failures.
/// The 3 methods `ready_transaction,get_descriptor,remove` are always used in conjuction so most likely
/// can be replaced with a single `drain()` method instead.
/// Thanks to this we also don't really need to lock the entire verification for the time of execution.
fn process_queue(&self) -> Result<(), Error> {
let nonce_cache = Default::default();
let mut verification_queue = self.transactions_for_verification.lock();
let ready_transactions = verification_queue.ready_transactions();
let fetch_nonce = |a: &Address| self.client.latest_nonce(a);
let ready_transactions = verification_queue.ready_transactions(self.pool_client(&nonce_cache));
for transaction in ready_transactions {
let transaction_hash = transaction.hash();
let transaction_hash = transaction.signed().hash();
match verification_queue.private_transaction_descriptor(&transaction_hash) {
Ok(desc) => {
if !self.validator_accounts.contains(&desc.validator_account) {
@@ -321,9 +327,10 @@ impl Provider where {
bail!(ErrorKind::ValidatorAccountNotSet);
}
let account = desc.validator_account;
if let Action::Call(contract) = transaction.action {
if let Action::Call(contract) = transaction.signed().action {
// TODO [ToDr] Usage of BlockId::Latest
let contract_nonce = self.get_contract_nonce(&contract, BlockId::Latest)?;
let private_state = self.execute_private_transaction(BlockId::Latest, &transaction)?;
let private_state = self.execute_private_transaction(BlockId::Latest, transaction.signed())?;
let private_state_hash = self.calculate_state_hash(&private_state, contract_nonce);
trace!("Hashed effective private state for validator: {:?}", private_state_hash);
let password = find_account_password(&self.passwords, &*self.accounts, &account);
@@ -341,7 +348,7 @@ impl Provider where {
bail!(e);
}
}
verification_queue.remove_private_transaction(&transaction_hash, &fetch_nonce);
verification_queue.remove_private_transaction(&transaction_hash);
}
Ok(())
}
@@ -354,6 +361,8 @@ impl Provider where {
let private_hash = tx.private_transaction_hash();
let desc = match self.transactions_for_signing.lock().get(&private_hash) {
None => {
// TODO [ToDr] Verification (we can't just blindly forward every transaction)
// Not our transaction, broadcast further to peers
self.broadcast_signed_private_transaction(rlp.into());
return Ok(());
@@ -383,7 +392,7 @@ impl Provider where {
let password = find_account_password(&self.passwords, &*self.accounts, &signer_account);
let signature = self.accounts.sign(signer_account, password, hash)?;
let signed = SignedTransaction::new(public_tx.with_signature(signature, chain_id))?;
match self.client.miner().import_own_transaction(&*self.client, signed.into()) {
match self.miner.import_own_transaction(&*self.client, signed.into()) {
Ok(_) => trace!("Public transaction added to queue"),
Err(err) => {
trace!("Failed to add transaction to queue, error: {:?}", err);

105
ethcore/private-tx/src/private_transactions.rs
View File

@@ -14,15 +14,16 @@
// You should have received a copy of the GNU General Public License
// along with Parity. If not, see <http://www.gnu.org/licenses/>.
use ethkey::Signature;
use std::sync::Arc;
use std::collections::{HashMap, HashSet};
use bytes::Bytes;
use std::collections::HashMap;
use ethcore_miner::pool;
use ethereum_types::{H256, U256, Address};
use ethkey::Signature;
use transaction::{UnverifiedTransaction, SignedTransaction};
use ethcore_miner::transaction_queue::{TransactionQueue, RemovalReason,
TransactionDetailsProvider as TransactionQueueDetailsProvider, TransactionOrigin};
use error::{Error, ErrorKind};
use ethcore::header::BlockNumber;
/// Maximum length for private transactions queues.
const MAX_QUEUE_LEN: usize = 8312;
@@ -39,56 +40,92 @@ pub struct PrivateTransactionDesc {
}
/// Storage for private transactions for verification
#[derive(Default)]
pub struct VerificationStore {
/// Descriptors for private transactions in queue for verification with key - hash of the original transaction
descriptors: HashMap<H256, PrivateTransactionDesc>,
/// Queue with transactions for verification
transactions: TransactionQueue,
///
/// TODO [ToDr] Might actually be better to use `txpool` directly and:
/// 1. Store descriptors inside `VerifiedTransaction`
/// 2. Use custom `ready` implementation to only fetch one transaction per sender.
/// 3. Get rid of passing dummy `block_number` and `timestamp`
transactions: pool::TransactionQueue,
}
impl Default for VerificationStore {
fn default() -> Self {
VerificationStore {
descriptors: Default::default(),
transactions: pool::TransactionQueue::new(
pool::Options {
max_count: MAX_QUEUE_LEN,
max_per_sender: MAX_QUEUE_LEN / 10,
max_mem_usage: 8 * 1024 * 1024,
},
pool::verifier::Options {
// TODO [ToDr] This should probably be based on some real values?
minimal_gas_price: 0.into(),
block_gas_limit: 8_000_000.into(),
tx_gas_limit: U256::max_value(),
},
pool::PrioritizationStrategy::GasPriceOnly,
)
}
}
}
impl VerificationStore {
/// Adds private transaction for verification into the store
pub fn add_transaction(
pub fn add_transaction<C: pool::client::Client>(
&mut self,
transaction: UnverifiedTransaction,
contract: Address,
validator_account: Address,
private_hash: H256,
details_provider: &TransactionQueueDetailsProvider,
insertion_time: BlockNumber,
client: C,
) -> Result<(), Error> {
if self.descriptors.len() > MAX_QUEUE_LEN {
bail!(ErrorKind::QueueIsFull);
}
if self.descriptors.get(&transaction.hash()).is_some() {
let transaction_hash = transaction.hash();
if self.descriptors.get(&transaction_hash).is_some() {
bail!(ErrorKind::PrivateTransactionAlreadyImported);
}
let transaction_hash = transaction.hash();
let signed_transaction = SignedTransaction::new(transaction)?;
self.transactions
.add(signed_transaction, TransactionOrigin::External, insertion_time, None, details_provider)
.and_then(|_| {
self.descriptors.insert(transaction_hash, PrivateTransactionDesc{
private_hash,
contract,
validator_account,
});
Ok(())
})
.map_err(Into::into)
let results = self.transactions.import(
client,
vec![pool::verifier::Transaction::Unverified(transaction)],
);
// Verify that transaction was imported
results.into_iter()
.next()
.expect("One transaction inserted; one result returned; qed")?;
self.descriptors.insert(transaction_hash, PrivateTransactionDesc {
private_hash,
contract,
validator_account,
});
Ok(())
}
/// Returns transactions ready for verification
/// Returns only one transaction per sender because several cannot be verified in a row without verification from other peers
pub fn ready_transactions(&self) -> Vec<SignedTransaction> {
// TODO [ToDr] Performance killer, re-work with new transaction queue.
let mut transactions = self.transactions.top_transactions();
// TODO [ToDr] Potential issue (create low address to have your transactions processed first)
transactions.sort_by(|a, b| a.sender().cmp(&b.sender()));
transactions.dedup_by(|a, b| a.sender().eq(&b.sender()));
transactions
pub fn ready_transactions<C: pool::client::NonceClient>(&self, client: C) -> Vec<Arc<pool::VerifiedTransaction>> {
// We never store PendingTransactions and we don't use internal cache,
// so we don't need to provide real block number of timestamp here
let block_number = 0;
let timestamp = 0;
let nonce_cap = None;
self.transactions.collect_pending(client, block_number, timestamp, nonce_cap, |transactions| {
// take only one transaction per sender
let mut senders = HashSet::with_capacity(self.descriptors.len());
transactions.filter(move |tx| senders.insert(tx.signed().sender())).collect()
})
}
/// Returns descriptor of the corresponding private transaction
@@ -97,11 +134,9 @@ impl VerificationStore {
}
/// Remove transaction from the queue for verification
pub fn remove_private_transaction<F>(&mut self, transaction_hash: &H256, fetch_nonce: &F)
where F: Fn(&Address) -> U256 {
pub fn remove_private_transaction(&mut self, transaction_hash: &H256) {
self.descriptors.remove(transaction_hash);
self.transactions.remove(transaction_hash, fetch_nonce, RemovalReason::Invalid);
self.transactions.remove(&[*transaction_hash], true);
}
}