Commit Graph

113 Commits

Author SHA1 Message Date
Robert Habermeier 18153b8d9e Merge branch 'master' into missing-code-by-prefix 2016-11-14 14:02:19 +01:00
Robert Habermeier 37f49aac1b abort snapshot restoration faster (#3356)
* abort snapshot restoration faster

* flag-checking tests
2016-11-13 13:52:53 +01:00
Gav Wood a7ad75d851 whitespace
[ci:skip]
2016-11-11 17:32:54 +01:00
Robert Habermeier ed135bb9dc fetch known code from the database during restoration
previously it kept all seen code in memory, leading to
high memory usage by the end of state restoration
2016-11-11 17:18:31 +01:00
Robert Habermeier 2806f1d4c9 Even more snapshot validity checks (#2935)
* clarify "cancelled periodic snapshot" message

* more rigorous checks for snapshot validity

* verify ancient blocks on import

* limit number of fed blocks

* make it possible to feed snapshot service canonical hashes

* fix failing test build

* swap ethash DAG only when more recent
2016-10-28 16:10:30 +02:00
Tomasz Drwięga 88997801d0 Clippy bump (#2877)
* Bumping clippy

* Fixing warnings

* Fix the "fix"
2016-10-27 08:28:12 +02:00
Arkadiy Paronyan f024acd329 More snapshot fixes and optimizations (#2883)
* More snapshot fixes and optimizations

* db drop
2016-10-26 16:14:13 +02:00
Arkadiy Paronyan 5b978be034 Get rid of MemoryDB denote (#2881) 2016-10-26 13:53:47 +02:00
Arkadiy Paronyan 135d5d0e4c Snapshot fixes and optimizations (#2863) 2016-10-25 18:40:01 +02:00
Robert Habermeier bc81ae0407 Snapshot and blockchain stability improvements (#2843)
* allow taking snapshot from just-restored database without error

* make creation informant less spammy

* Ancestry iterator failure-resilient

* make uncle hash searching resilient to incomplete chain

* deduce pre-chunk info from last written block's details
2016-10-24 18:27:23 +02:00
Arkadiy Paronyan ae853a7557 Don't add empty accounts to bloom (#2753) 2016-10-20 16:49:27 +02:00
Arkadiy Paronyan 487dfb0208 Snapshot sync part 2 (#2098)
* Split block downloader into a module

* Snapshot sync progress

* Warp sync CLI option

* Increased snapshot chunk and ping timeouts

* Fixed an issue with delayed writes

* Updated bootnodes

* Don't run pending IO tasks on shutdown

* Optional first_block; removed insert_snapshot_block

* Fixing expect calls

* Fixed stalled sync

* style and docs

* Update block_sync.rs

[ci:skip]
2016-10-18 18:16:00 +02:00
Nikolay Volf 7526b1d44b Derive IPC interface only when ipc feature is on (#2463)
* derive -> ipc

* accident repair

* conditional ipc deriving

* fix test
2016-10-05 19:42:57 +02:00
Nikolay Volf e1d3b3fff8 Accounts bloom in master (#2426)
* bloom crate link

* database layout and outdated tests

* state db alterations

* v10 migration run

* using arc

* bloom migration

* migration fixes and mess

* fix tests
2016-10-03 12:02:43 +02:00
Robert Habermeier 723d837d05 Snapshot format changes (#2234)
Closes #2213
Omit transaction and receipt roots from abridged block.
No longer use RLP compression.

Make ordered_trie_root generic over an iterator to save an allocation.

Breaks snapshot format backwards compatibility (with other 1.4 snapshots -- it's already been broken with 1.3).

Documentation will need updating
2016-09-22 19:47:03 +02:00
Robert Habermeier a100b9d09e Remove panickers from trie iterators (#2209)
* port trie iterators to use error handling

* use new trie iterators in snapshot

allows proper recovery from a premature periodic snapshot
2016-09-21 12:56:13 +02:00
Robert Habermeier f054a7b8d5 more info on current periodic snapshot 2016-09-06 17:44:11 +02:00
Robert Habermeier 1e2cdedc9b merge with master 2016-09-06 15:41:56 +02:00
Arkadiy Paronyan 5c5d9c8ccd Snapshot sync (#2047)
* PV64 sync

* Tests

* Client DB restore

* Snapshot restoration over IPC

* Upating test

* Minor tweaks

* Upating test
2016-09-06 15:31:13 +02:00
Robert Habermeier 46b1224f7c Merge branch 'master' into periodic_snapshot 2016-09-05 14:50:43 +02:00
Robert Habermeier 4e466f09db Add a few small snapshot tests (#2038)
* add manifest data test

* begin tests for io

* add unit tests for snapshot io

* service smoke test

* remove empty service module
2016-09-05 12:24:03 +02:00
Robert Habermeier e3749b3bc4 implement snapshot watcher 2016-09-02 18:28:47 +02:00
Robert Habermeier 5dd56aa070 port ethcore 2016-09-01 14:29:59 +02:00
Robert Habermeier 1c19a807d9 Take control of recovered snapshots, start restoration asynchronously (#2010)
* take control of given snapshot

* start snapshot restoration asynchronously,
2016-08-25 22:20:44 +02:00
Robert Habermeier 2aef81cf90 remove internal locking from DBTransaction (#2003) 2016-08-25 16:43:56 +02:00
Robert Habermeier b18407b9e3 Snapshot optimizations (#1991)
* apply RLP compression to abridged blocks

* add memorydb consolidate

* code hash optimization

* add warning to snapshot restoration CLI
2016-08-25 14:28:45 +02:00
Robert Habermeier 190e4db266 Trie query recording and AccountDB factory for no mangling (#1944)
* optionally use no mangling for accountdb

* add the recorder module

* get_recorded for tries, no virtual dispatch on readonly tries

* add recording test
2016-08-24 16:53:36 +02:00
Robert Habermeier 9adf77581c move column family constants into db module (#1955) 2016-08-18 18:24:49 +02:00
Tomasz Drwięga a427208f79 Purging .derefs, fixing clippy warnings. (#1890)
* Fixing clippy warnings

* Purging derefs

* Simplifying engine derefs

* Simplifying more engine derefs
2016-08-10 16:29:40 +02:00
Robert Habermeier 2f1ade8116 take snapshot at specified block and slightly better informants (#1873)
* prettier informant for snapshot creation

* allow taking snapshot at a given block

* minor tweaks

* elaborate on cli
2016-08-08 18:41:30 +02:00
Robert Habermeier 76a7246369 Snapshot creation and restoration (#1679)
* to_rlp takes self by-reference

* clean up some derefs

* out-of-order insertion for blockchain

* implement block rebuilder without verification

* group block chunk header into struct

* block rebuilder does verification

* integrate snapshot service with client service; flesh out implementation more

* initial implementation of snapshot service

* remove snapshottaker trait

* snapshot writer trait with packed and loose implementations

* write chunks using "snapshotwriter" in service

* have snapshot taking use snapshotwriter

* implement snapshot readers

* back up client dbs when replacing

* use snapshot reader in snapshot service

* describe offset format

* use new get_db_path in parity, allow some errors in service

* blockchain formatting

* implement parity snapshot

* implement snapshot restore

* force blocks to be submitted in order

* fix bug loading block hashes in packed reader

* fix seal field loading

* fix uncle hash computation

* fix a few bugs

* store genesis state in db. reverse block chunk order in packed writer

* allow out-of-order import for blocks

* bring restoration types together

* only snapshot the last 30000 blocks

* restore into overlaydb instead of journaldb

* commit version to database

* use memorydbs and commit directly

* fix trie test compilation

* fix failing tests

* sha3_null_rlp, not H256::zero

* move overlaydb to ref_overlaydb, add new overlaydb without on-disk rc

* port archivedb to new overlaydb

* add deletion mode tests for overlaydb

* use new overlaydb, check state root at end

* share chain info between state and block snapshotting

* create blocks snapshot using blockchain directly

* allow snapshot from arbitrary block, remove panickers from snapshot creation

* begin test framework

* blockchain chunking test

* implement stateproducer::tick

* state snapshot test

* create block and state chunks concurrently, better restoration informant

* fix tests

* add deletion mode tests for overlaydb

* address comments

* more tests

* Fix up tests.

* remove a few printlns

* add a little more documentation to `commit`

* fix tests

* fix ref_overlaydb test names

* snapshot command skeleton

* revert ref_overlaydb renaming

* reimplement snapshot commands

* fix many errors

* everything but inject

* get ethcore compiling

* get snapshot tests passing again

* instrument snapshot commands again

* fix fallout from other changes, mark snapshots as experimental

* optimize injection patterns

* do two injections

* fix up tests

* take snapshots from 1000 blocks efore

* address minor comments

* fix a few io crate related errors

* clarify names about total difficulty

[ci skip]
2016-08-05 17:00:46 +02:00
Tomasz Drwięga deceb5fd56 Bumping clippy & fixing warnings (#1823)
* Bumping clippy

* Fixing clippy warnings

* Cargo.lock (latest nightly?)
2016-08-03 19:01:48 +02:00
Robert Habermeier 11b65ce53d Remove (almost all) panickers from trie module (#1776)
* memorydb ub patch and other cleanup

* fix denote invocations

* move trie traits into trie module

* replace "denote" with shim

* triedb returns results and no longer panics

* fix warnings

* get ethcore compiling

* warn on trie errors in ethcore

* remove unsafety from node decoder

* restore broken denote behavior for this branch

* fix overlayrecent fallout

* fix triedb tests

* remove unwrap in state

* alter Trie::get to return Result<Option<_>>

* fix refcell error in require

* fix test warnings

* fix json tests

* whitespace

[ci:skip]

* Avoid unneeded match/indentation

* whitespace

* prettify map_or_else

* remove test warning
2016-08-03 18:35:48 +02:00
Tomasz Drwięga e4f0c0b215 Single DB (#1741)
* Consolidation migration

* Started db amalgamation

* Using client constants for columns

* Adding with_columns constructor

* Migrating to single db

* Fixing tests.

* test.sh without verbose

* Fixing warnings

* add migration tests that catch the bug

* make multiple migrations more robust

* add moved v9

* Merge branch 'noop-migrations' into single-db

* spurious line

* clean up migrations ordering

* update comment [ci skip]

* Bumping default number of max_open_files & re-ordering columns.

* fix merge

* fix ignored analysis tests

* Caching best block content

* Faster best_block_header

* Adding progress to v8 migration

* clean up warnings

* Separate hashes and bodies in the DB

* Separate hashes and bodies in the DB

* Fixed tests
2016-07-28 23:46:24 +02:00
keorn 02cf48681d Blocks and snapshot compression (#1687)
* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* new Compressible rlp trait

* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* DecompressingDecoder test

* initial compressing HashDB wrapper

* remove unused test

* change CompressedDB to struct wrapper with overlay

* simplify compressor

* failed RefCell attempt

* use denote to return reference

* compiled compresseddb

* compressdb test, add overlay emplace

* fix overlay reference count handling

* add immutable compresseddb, make account use hashdb

* simplify using trait objects

* enable hashdb for account

* initial state compression attempt

* wrap state db

* add tests for analyzing db

* add account predicate

* try to compress data fields as rlp too

* remove compression for storage trie

* add a compressing migration

* more compression stats tests

* fix migration import

* nested encoding compression test

* fix decompression, move db stats tests to rlpcompression

* added malformed rlp tests, cover a few edge cases

* new CompressingEncoder struct

* extend migrations to state

* first version working on the whole db

* clean up Compressible impl

* tests cleanup

* add a testing migration

* refactor deep compression using option, add simple compression

* put tests in a module

* fix compressed overlay loading

* simple compression for snapshots

* remove unused DecompressingDecoder

* add a general compressing migration

* add more common rlps to compress

* use static slices for swapper

* add precomputed hashes and invalid rlps

* make decoder private again

* cover more cases with tests

* style

* fix weird indentation

* remove possible panic in payload_info

* make prefix checking safe

* fix db existence check

* remove db dir from test

* pass usize by value [ci skip]

* Improve comment on panic removal.

* add common blocks db rlps

* add compression to blockchain db

* add blocks db migration

* fix the migrations

* remove state compression

* add a separate snapshot swapper

* ability to use different swappers and traversal

* update tests to new interface

* clean up code ordering

* update usage

* fix compilation

* remove unnecessary changes

* move methods to functions to reduce interface

* move test to module

* update common rlps to blocks db

* move tests to tests modules

* remove redundant &
2016-07-27 17:11:41 +02:00
Tomasz Drwięga 038862fa9d Fixing clippy warnings (#1660) 2016-07-19 09:25:51 +02:00
Robert Habermeier 4269867ca4 remove unnecessary assertion 2016-07-11 19:56:27 +02:00
Robert Habermeier 49ba117f13 add account tests 2016-07-11 16:54:50 +02:00
Robert Habermeier da9e9692dc remove outdated comment; prove panickers; update manifest 2016-07-11 13:17:26 +02:00
Robert Habermeier 36dfa4743e merge accountdb migration 2016-07-09 17:33:14 +02:00
Robert Habermeier 456619001a parallelize account trie creation 2016-06-30 20:43:54 +02:00
Robert Habermeier 6ecd6eaa12 batch state trie writes 2016-06-29 18:37:17 +02:00
Robert Habermeier 3850ee64bb have StateRebuilder take a JournalDB and commit post-chunk 2016-06-17 12:56:57 +02:00
Robert Habermeier e4867d7cb9 properly rebuild state trie 2016-06-16 18:30:18 +02:00
Robert Habermeier f45254d485 create chunks using append_raw 2016-06-16 16:39:42 +02:00
Robert Habermeier 0e917b9fa3 use ZST error type for snappy, use new snappy compression methods 2016-06-16 15:35:49 +02:00
Robert Habermeier 7ca5c115d8 validate all compressed buffers 2016-06-15 19:18:49 +02:00
Robert Habermeier 25f1f22c86 use account_pair in feed 2016-06-15 18:41:02 +02:00
Robert Habermeier 0e3a15cadb add a state rebuilder 2016-06-15 17:46:40 +02:00
Robert Habermeier d7498c1dd5 rename AccountReader to Account, give a separate module 2016-06-15 16:42:49 +02:00
Robert Habermeier a2bb3f2832 account storage decoding implemented 2016-06-15 16:34:11 +02:00
Robert Habermeier d26e038dc7 fix bug in account code storage, add to_thin_rlp for AccountReader 2016-06-15 15:59:21 +02:00
Robert Habermeier 9badb310ae remove snappy_buffer_size constant in favor of max_compressed_len 2016-06-15 13:05:00 +02:00
Robert Habermeier 17a60920cb adjust comments 2016-06-14 19:14:41 +02:00
Robert Habermeier 3290f393bd block rlp compression 2016-06-14 18:34:27 +02:00
Robert Habermeier 8ed2b9c7d9 refactor out common parts of write_chunk 2016-06-14 15:32:49 +02:00
Robert Habermeier dfb603dd08 compress into reusable buffers 2016-06-14 13:22:15 +02:00
Robert Habermeier 16e58958c9 use snappy compression on buffers 2016-06-14 12:32:51 +02:00
Robert Habermeier b88eef5374 use append_empty_data rather than fibbing it 2016-06-14 11:48:34 +02:00
Robert Habermeier 835b158069 store account code if it exists 2016-06-13 21:05:06 +02:00
Robert Habermeier 5cc6a681c9 push pair onto block chunker buffer 2016-06-13 20:48:12 +02:00
Robert Habermeier c36f47bbbf rename pv64_snapshot trace target to snapshot 2016-06-13 20:00:47 +02:00
Robert Habermeier 22a19819c6 rename pv64 module to snapshot 2016-06-13 20:00:00 +02:00