Snapshot creation and restoration (#1679)

* to_rlp takes self by-reference

* clean up some derefs

* out-of-order insertion for blockchain

* implement block rebuilder without verification

* group block chunk header into struct

* block rebuilder does verification

* integrate snapshot service with client service; flesh out implementation more

* initial implementation of snapshot service

* remove snapshottaker trait

* snapshot writer trait with packed and loose implementations

* write chunks using "snapshotwriter" in service

* have snapshot taking use snapshotwriter

* implement snapshot readers

* back up client dbs when replacing

* use snapshot reader in snapshot service

* describe offset format

* use new get_db_path in parity, allow some errors in service

* blockchain formatting

* implement parity snapshot

* implement snapshot restore

* force blocks to be submitted in order

* fix bug loading block hashes in packed reader

* fix seal field loading

* fix uncle hash computation

* fix a few bugs

* store genesis state in db. reverse block chunk order in packed writer

* allow out-of-order import for blocks

* bring restoration types together

* only snapshot the last 30000 blocks

* restore into overlaydb instead of journaldb

* commit version to database

* use memorydbs and commit directly

* fix trie test compilation

* fix failing tests

* sha3_null_rlp, not H256::zero

* move overlaydb to ref_overlaydb, add new overlaydb without on-disk rc

* port archivedb to new overlaydb

* add deletion mode tests for overlaydb

* use new overlaydb, check state root at end

* share chain info between state and block snapshotting

* create blocks snapshot using blockchain directly

* allow snapshot from arbitrary block, remove panickers from snapshot creation

* begin test framework

* blockchain chunking test

* implement stateproducer::tick

* state snapshot test

* create block and state chunks concurrently, better restoration informant

* fix tests

* add deletion mode tests for overlaydb

* address comments

* more tests

* Fix up tests.

* remove a few printlns

* add a little more documentation to `commit`

* fix tests

* fix ref_overlaydb test names

* snapshot command skeleton

* revert ref_overlaydb renaming

* reimplement snapshot commands

* fix many errors

* everything but inject

* get ethcore compiling

* get snapshot tests passing again

* instrument snapshot commands again

* fix fallout from other changes, mark snapshots as experimental

* optimize injection patterns

* do two injections

* fix up tests

* take snapshots from 1000 blocks efore

* address minor comments

* fix a few io crate related errors

* clarify names about total difficulty

[ci skip]

util/src/journaldb/mod.rs

@@ -184,4 +184,4 @@ mod tests {
 		assert_eq!(overlayrecent, 1);
 		assert_eq!(refcounted, 1);
 	}
-}
+}

util/src/journaldb/refcounteddb.rs

@@ -19,7 +19,7 @@
 use common::*;
 use rlp::*;
 use hashdb::*;
-use overlaydb::*;
+use overlaydb::OverlayDB;
 use super::{DB_PREFIX_LEN, LATEST_ERA_KEY};
 use super::traits::JournalDB;
 use kvdb::{Database, DBTransaction};

util/src/memorydb.rs

@@ -134,7 +134,7 @@ impl MemoryDB {
 		if key == &SHA3_NULL_RLP {
 			return Some(STATIC_NULL_RLP.clone());
 		}
-		self.data.get(key).map(|&(ref v, x)| (&v[..], x))
+		self.data.get(key).map(|&(ref val, rc)| (&val[..], rc))
 	}
 
 	/// Denote than an existing value has the given key. Used when a key gets removed without

util/src/overlaydb.rs

@@ -314,4 +314,4 @@ fn playpen() {
 		db.write(batch).unwrap();
 	}
 	fs::remove_dir_all("/tmp/test").unwrap();
-}
+}