Commit Graph

255 Commits

Author SHA1 Message Date
Robert Habermeier
37f49aac1b abort snapshot restoration faster (#3356)
* abort snapshot restoration faster

* flag-checking tests
2016-11-13 13:52:53 +01:00
Robert Habermeier
87c668ba9c test previous code fetch 2016-11-11 18:24:12 +01:00
Gav Wood
a7ad75d851 whitespace
[ci:skip]
2016-11-11 17:32:54 +01:00
Robert Habermeier
ed135bb9dc fetch known code from the database during restoration
previously it kept all seen code in memory, leading to
high memory usage by the end of state restoration
2016-11-11 17:18:31 +01:00
keorn
ea960f1c27 Merge remote-tracking branch 'parity/master' into bft 2016-11-07 11:34:45 +00:00
Arkadiy Paronyan
bccc56b6b0 mio version bump (#2982) 2016-10-30 09:56:34 +01:00
Tomasz Drwięga
f17cad9dff Fixing clippy warnings 2 (#2961) 2016-10-29 13:08:49 +02:00
Robert Habermeier
2806f1d4c9 Even more snapshot validity checks (#2935)
* clarify "cancelled periodic snapshot" message

* more rigorous checks for snapshot validity

* verify ancient blocks on import

* limit number of fed blocks

* make it possible to feed snapshot service canonical hashes

* fix failing test build

* swap ethash DAG only when more recent
2016-10-28 16:10:30 +02:00
Tomasz Drwięga
88997801d0 Clippy bump (#2877)
* Bumping clippy

* Fixing warnings

* Fix the "fix"
2016-10-27 08:28:12 +02:00
Arkadiy Paronyan
f024acd329 More snapshot fixes and optimizations (#2883)
* More snapshot fixes and optimizations

* db drop
2016-10-26 16:14:13 +02:00
Arkadiy Paronyan
5b978be034 Get rid of MemoryDB denote (#2881) 2016-10-26 13:53:47 +02:00
Arkadiy Paronyan
135d5d0e4c Snapshot fixes and optimizations (#2863) 2016-10-25 18:40:01 +02:00
Robert Habermeier
bc81ae0407 Snapshot and blockchain stability improvements (#2843)
* allow taking snapshot from just-restored database without error

* make creation informant less spammy

* Ancestry iterator failure-resilient

* make uncle hash searching resilient to incomplete chain

* deduce pre-chunk info from last written block's details
2016-10-24 18:27:23 +02:00
keorn
866ab9c7a3 Check queue to determine major importing (#2763)
* simplify major sync detection

* fix typos

* fix merge

* more realistic EthTester

* add new synced state

* remove Blocks synced state

* move is_major_importing to rpc crate and check queue

* add tests
2016-10-20 23:36:18 +02:00
Arkadiy Paronyan
ae853a7557 Don't add empty accounts to bloom (#2753) 2016-10-20 16:49:27 +02:00
keorn
aa52b04e31 Unify major syncing detection (#2699)
* simplify major sync detection

* fix typos

* fix merge

* more realistic EthTester

* add new synced state
2016-10-19 18:35:39 +02:00
Arkadiy Paronyan
487dfb0208 Snapshot sync part 2 (#2098)
* Split block downloader into a module

* Snapshot sync progress

* Warp sync CLI option

* Increased snapshot chunk and ping timeouts

* Fixed an issue with delayed writes

* Updated bootnodes

* Don't run pending IO tasks on shutdown

* Optional first_block; removed insert_snapshot_block

* Fixing expect calls

* Fixed stalled sync

* style and docs

* Update block_sync.rs

[ci:skip]
2016-10-18 18:16:00 +02:00
Robert Habermeier
8263bd4be2 snapshot: single byte for empty accounts (#2625) 2016-10-14 15:45:45 +03:00
Robert Habermeier
f28b8352c1 Configurable history size in master (#2606)
* split journaldb commit into two functions: journal_under and mark_canonical

* use new commit mechanism in client

* Configurable history size in master

* Reduce DB history

* Configurable history size

* Set min history size

* Test

* Fixed a test and reduced the limit
2016-10-14 15:44:56 +03:00
keorn
e343153f06 mixed merge and changes... 2016-10-11 18:37:31 +01:00
Nikolay Volf
7526b1d44b Derive IPC interface only when ipc feature is on (#2463)
* derive -> ipc

* accident repair

* conditional ipc deriving

* fix test
2016-10-05 19:42:57 +02:00
keorn
1f56588b87 Merge remote-tracking branch 'parity/master' into bft
Conflicts:
	ethcore/src/client/client.rs
2016-10-05 14:57:14 +01:00
keorn
67c24dcb95 use Engine to order blockchain 2016-10-05 14:29:35 +01:00
Nikolay Volf
2b147616fd derive -> ipc 2016-10-04 19:20:16 +03:00
Nikolay Volf
e1d3b3fff8 Accounts bloom in master (#2426)
* bloom crate link

* database layout and outdated tests

* state db alterations

* v10 migration run

* using arc

* bloom migration

* migration fixes and mess

* fix tests
2016-10-03 12:02:43 +02:00
Arkadiy Paronyan
ad63780b4d Canonical state cache (master) (#2311)
* State cache

* Reduced copying data between caches

Whitespace and optional symbols

* Reduced copying data between caches

Whitespace and optional symbols

* Set a limit on storage cache

* Style and docs
2016-09-27 18:02:11 +02:00
Robert Habermeier
723d837d05 Snapshot format changes (#2234)
Closes #2213
Omit transaction and receipt roots from abridged block.
No longer use RLP compression.

Make ordered_trie_root generic over an iterator to save an allocation.

Breaks snapshot format backwards compatibility (with other 1.4 snapshots -- it's already been broken with 1.3).

Documentation will need updating
2016-09-22 19:47:03 +02:00
Robert Habermeier
a100b9d09e Remove panickers from trie iterators (#2209)
* port trie iterators to use error handling

* use new trie iterators in snapshot

allows proper recovery from a premature periodic snapshot
2016-09-21 12:56:13 +02:00
Tomasz Drwięga
7f7e74869c Bump clippy & Fix warnings (#2109)
* Bump clippy

* Fixing warnings
2016-09-16 23:03:26 +02:00
Robert Habermeier
83ddce011d ensure the target folder doesn't exist before renaming (#2074) 2016-09-13 10:33:03 +02:00
Robert Habermeier
edcc4080d5 fix failing master build: update tests to new init_restore signature. (#2069) 2016-09-11 20:17:05 +02:00
Robert Habermeier
dcfd7eab6d Local snapshot restore (#2058)
* restore from local snapshot

* update status with chunks done

* rework local restore trigger
2016-09-11 14:05:59 +02:00
Robert Habermeier
ff17174cf1 also test abort_restore 2016-09-08 12:44:10 +02:00
Robert Habermeier
756b7a3e67 test folder-deleting guards 2016-09-08 12:27:13 +02:00
Robert Habermeier
c65a5c8e9c add restored_is_equivalent test 2016-09-08 12:14:53 +02:00
Robert Habermeier
b0f6bf2e78 remove client_db field from snapshot service 2016-09-07 19:38:59 +02:00
Robert Habermeier
57d5c35bb6 Use proper database configuration in snapshots. (#2052)
* use proper database config in snapshot service

* add snapshot path to parity directories struct

* fix RPC tests
2016-09-07 15:27:28 +02:00
Robert Habermeier
541b14a4ab periodic snapshot tweaks (#2054)
* periodic snapshot tweaks

* set SNAPSHOT_HISTORY to 500
2016-09-07 15:27:14 +02:00
Robert Habermeier
f054a7b8d5 more info on current periodic snapshot 2016-09-06 17:44:11 +02:00
Robert Habermeier
46581e173d check block queue size before taking periodic snapshot 2016-09-06 15:49:44 +02:00
Robert Habermeier
1e2cdedc9b merge with master 2016-09-06 15:41:56 +02:00
Arkadiy Paronyan
5c5d9c8ccd Snapshot sync (#2047)
* PV64 sync

* Tests

* Client DB restore

* Snapshot restoration over IPC

* Upating test

* Minor tweaks

* Upating test
2016-09-06 15:31:13 +02:00
Robert Habermeier
46b1224f7c Merge branch 'master' into periodic_snapshot 2016-09-05 14:50:43 +02:00
Robert Habermeier
f0ef5e6943 keep snapshot watcher alive 2016-09-05 14:25:56 +02:00
Robert Habermeier
4e466f09db Add a few small snapshot tests (#2038)
* add manifest data test

* begin tests for io

* add unit tests for snapshot io

* service smoke test

* remove empty service module
2016-09-05 12:24:03 +02:00
Robert Habermeier
2bf235e226 use more mocking in tests 2016-09-05 12:17:21 +02:00
Robert Habermeier
d9eb87cae7 add guard for temporary directories 2016-09-02 19:00:20 +02:00
Robert Habermeier
e3749b3bc4 implement snapshot watcher 2016-09-02 18:28:47 +02:00
Robert Habermeier
04dee54cb3 add take_snapshot to snapshot service 2016-09-02 16:15:25 +02:00
Robert Habermeier
5dd56aa070 port ethcore 2016-09-01 14:29:59 +02:00
Nipunn Koorapati
4389742ca3 Make the block header struct's internals private (#2000)
* Make the block header struct's internals private

Currently, this involves a lot of explicit cloning, but we
could migrate the return types of the get_* functions to
be copies rather than references since they are mostly copy
types anyway.

I opted to eliminate the constructor in favor of using
Default::default() plus calling a bunch of setters. This
is similar to the model that a Google Protobuf client uses
and I think it looks fine.

* Drop some unnecessary cloning by comparing references

* Fix compiler errors from callsites in tests.
2016-08-29 11:35:23 +02:00
Robert Habermeier
1c19a807d9 Take control of recovered snapshots, start restoration asynchronously (#2010)
* take control of given snapshot

* start snapshot restoration asynchronously,
2016-08-25 22:20:44 +02:00
Robert Habermeier
2aef81cf90 remove internal locking from DBTransaction (#2003) 2016-08-25 16:43:56 +02:00
Robert Habermeier
b18407b9e3 Snapshot optimizations (#1991)
* apply RLP compression to abridged blocks

* add memorydb consolidate

* code hash optimization

* add warning to snapshot restoration CLI
2016-08-25 14:28:45 +02:00
Robert Habermeier
190e4db266 Trie query recording and AccountDB factory for no mangling (#1944)
* optionally use no mangling for accountdb

* add the recorder module

* get_recorded for tries, no virtual dispatch on readonly tries

* add recording test
2016-08-24 16:53:36 +02:00
Nikolay Volf
d7c184bd72 Merge pull request #1953 from ethcore/state_module
introduce ethcore/state module
2016-08-18 21:56:08 +04:00
Robert Habermeier
9adf77581c move column family constants into db module (#1955) 2016-08-18 18:24:49 +02:00
Robert Habermeier
eed7d006f1 Fix up tests, remove dead state-diff code 2016-08-18 18:13:32 +02:00
Marek Kotewicz
c39761c042 ethkey and ethstore use hash structures from bigint (#1851)
* Address renamed to H160 at bigint library level

* moved uint specific test from util to bigint library

* naming

* unifing hashes in progress

* unifing hashes

* cleanup redundant unwraps in tests

* fixed compiling
2016-08-15 15:09:00 +02:00
Tomasz Drwięga
a427208f79 Purging .derefs, fixing clippy warnings. (#1890)
* Fixing clippy warnings

* Purging derefs

* Simplifying engine derefs

* Simplifying more engine derefs
2016-08-10 16:29:40 +02:00
debris
974d537849 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-09 09:58:18 +02:00
Robert Habermeier
2f1ade8116 take snapshot at specified block and slightly better informants (#1873)
* prettier informant for snapshot creation

* allow taking snapshot at a given block

* minor tweaks

* elaborate on cli
2016-08-08 18:41:30 +02:00
debris
88c5f555a9 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-06 00:03:07 +02:00
Robert Habermeier
e72fc5398a miner and client take spec reference (#1853)
* miner and client take spec reference

* fix tests
2016-08-05 23:33:55 +02:00
Robert Habermeier
76a7246369 Snapshot creation and restoration (#1679)
* to_rlp takes self by-reference

* clean up some derefs

* out-of-order insertion for blockchain

* implement block rebuilder without verification

* group block chunk header into struct

* block rebuilder does verification

* integrate snapshot service with client service; flesh out implementation more

* initial implementation of snapshot service

* remove snapshottaker trait

* snapshot writer trait with packed and loose implementations

* write chunks using "snapshotwriter" in service

* have snapshot taking use snapshotwriter

* implement snapshot readers

* back up client dbs when replacing

* use snapshot reader in snapshot service

* describe offset format

* use new get_db_path in parity, allow some errors in service

* blockchain formatting

* implement parity snapshot

* implement snapshot restore

* force blocks to be submitted in order

* fix bug loading block hashes in packed reader

* fix seal field loading

* fix uncle hash computation

* fix a few bugs

* store genesis state in db. reverse block chunk order in packed writer

* allow out-of-order import for blocks

* bring restoration types together

* only snapshot the last 30000 blocks

* restore into overlaydb instead of journaldb

* commit version to database

* use memorydbs and commit directly

* fix trie test compilation

* fix failing tests

* sha3_null_rlp, not H256::zero

* move overlaydb to ref_overlaydb, add new overlaydb without on-disk rc

* port archivedb to new overlaydb

* add deletion mode tests for overlaydb

* use new overlaydb, check state root at end

* share chain info between state and block snapshotting

* create blocks snapshot using blockchain directly

* allow snapshot from arbitrary block, remove panickers from snapshot creation

* begin test framework

* blockchain chunking test

* implement stateproducer::tick

* state snapshot test

* create block and state chunks concurrently, better restoration informant

* fix tests

* add deletion mode tests for overlaydb

* address comments

* more tests

* Fix up tests.

* remove a few printlns

* add a little more documentation to `commit`

* fix tests

* fix ref_overlaydb test names

* snapshot command skeleton

* revert ref_overlaydb renaming

* reimplement snapshot commands

* fix many errors

* everything but inject

* get ethcore compiling

* get snapshot tests passing again

* instrument snapshot commands again

* fix fallout from other changes, mark snapshots as experimental

* optimize injection patterns

* do two injections

* fix up tests

* take snapshots from 1000 blocks efore

* address minor comments

* fix a few io crate related errors

* clarify names about total difficulty

[ci skip]
2016-08-05 17:00:46 +02:00
debris
03c3d16744 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-04 08:41:30 +02:00
Tomasz Drwięga
deceb5fd56 Bumping clippy & fixing warnings (#1823)
* Bumping clippy

* Fixing clippy warnings

* Cargo.lock (latest nightly?)
2016-08-03 19:01:48 +02:00
Robert Habermeier
11b65ce53d Remove (almost all) panickers from trie module (#1776)
* memorydb ub patch and other cleanup

* fix denote invocations

* move trie traits into trie module

* replace "denote" with shim

* triedb returns results and no longer panics

* fix warnings

* get ethcore compiling

* warn on trie errors in ethcore

* remove unsafety from node decoder

* restore broken denote behavior for this branch

* fix overlayrecent fallout

* fix triedb tests

* remove unwrap in state

* alter Trie::get to return Result<Option<_>>

* fix refcell error in require

* fix test warnings

* fix json tests

* whitespace

[ci:skip]

* Avoid unneeded match/indentation

* whitespace

* prettify map_or_else

* remove test warning
2016-08-03 18:35:48 +02:00
debris
e8c451ac82 cleaning up hash reexports 2016-08-03 18:05:17 +02:00
Tomasz Drwięga
e4f0c0b215 Single DB (#1741)
* Consolidation migration

* Started db amalgamation

* Using client constants for columns

* Adding with_columns constructor

* Migrating to single db

* Fixing tests.

* test.sh without verbose

* Fixing warnings

* add migration tests that catch the bug

* make multiple migrations more robust

* add moved v9

* Merge branch 'noop-migrations' into single-db

* spurious line

* clean up migrations ordering

* update comment [ci skip]

* Bumping default number of max_open_files & re-ordering columns.

* fix merge

* fix ignored analysis tests

* Caching best block content

* Faster best_block_header

* Adding progress to v8 migration

* clean up warnings

* Separate hashes and bodies in the DB

* Separate hashes and bodies in the DB

* Fixed tests
2016-07-28 23:46:24 +02:00
keorn
02cf48681d Blocks and snapshot compression (#1687)
* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* new Compressible rlp trait

* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* DecompressingDecoder test

* initial compressing HashDB wrapper

* remove unused test

* change CompressedDB to struct wrapper with overlay

* simplify compressor

* failed RefCell attempt

* use denote to return reference

* compiled compresseddb

* compressdb test, add overlay emplace

* fix overlay reference count handling

* add immutable compresseddb, make account use hashdb

* simplify using trait objects

* enable hashdb for account

* initial state compression attempt

* wrap state db

* add tests for analyzing db

* add account predicate

* try to compress data fields as rlp too

* remove compression for storage trie

* add a compressing migration

* more compression stats tests

* fix migration import

* nested encoding compression test

* fix decompression, move db stats tests to rlpcompression

* added malformed rlp tests, cover a few edge cases

* new CompressingEncoder struct

* extend migrations to state

* first version working on the whole db

* clean up Compressible impl

* tests cleanup

* add a testing migration

* refactor deep compression using option, add simple compression

* put tests in a module

* fix compressed overlay loading

* simple compression for snapshots

* remove unused DecompressingDecoder

* add a general compressing migration

* add more common rlps to compress

* use static slices for swapper

* add precomputed hashes and invalid rlps

* make decoder private again

* cover more cases with tests

* style

* fix weird indentation

* remove possible panic in payload_info

* make prefix checking safe

* fix db existence check

* remove db dir from test

* pass usize by value [ci skip]

* Improve comment on panic removal.

* add common blocks db rlps

* add compression to blockchain db

* add blocks db migration

* fix the migrations

* remove state compression

* add a separate snapshot swapper

* ability to use different swappers and traversal

* update tests to new interface

* clean up code ordering

* update usage

* fix compilation

* remove unnecessary changes

* move methods to functions to reduce interface

* move test to module

* update common rlps to blocks db

* move tests to tests modules

* remove redundant &
2016-07-27 17:11:41 +02:00
Tomasz Drwięga
038862fa9d Fixing clippy warnings (#1660) 2016-07-19 09:25:51 +02:00
Robert Habermeier
4269867ca4 remove unnecessary assertion 2016-07-11 19:56:27 +02:00
Robert Habermeier
673cde6c7f add block tests 2016-07-11 18:31:18 +02:00
Robert Habermeier
49ba117f13 add account tests 2016-07-11 16:54:50 +02:00
Robert Habermeier
24ec696062 rebuild uncles hash from uncle headers 2016-07-11 15:19:01 +02:00
Robert Habermeier
da9e9692dc remove outdated comment; prove panickers; update manifest 2016-07-11 13:17:26 +02:00
Robert Habermeier
c7af950992 ignore warnings temporarily 2016-07-11 11:10:08 +02:00
Robert Habermeier
36dfa4743e merge accountdb migration 2016-07-09 17:33:14 +02:00
Robert Habermeier
456619001a parallelize account trie creation 2016-06-30 20:43:54 +02:00
Robert Habermeier
6ecd6eaa12 batch state trie writes 2016-06-29 18:37:17 +02:00
Robert Habermeier
49024a4f28 Merge with master 2016-06-29 14:46:29 +02:00
Robert Habermeier
3850ee64bb have StateRebuilder take a JournalDB and commit post-chunk 2016-06-17 12:56:57 +02:00
Robert Habermeier
f3c1643090 encode the account storage rlp with "append_raw" 2016-06-16 19:47:05 +02:00
Robert Habermeier
e4867d7cb9 properly rebuild state trie 2016-06-16 18:30:18 +02:00
Robert Habermeier
f45254d485 create chunks using append_raw 2016-06-16 16:39:42 +02:00
Robert Habermeier
0e917b9fa3 use ZST error type for snappy, use new snappy compression methods 2016-06-16 15:35:49 +02:00
Robert Habermeier
7ca5c115d8 validate all compressed buffers 2016-06-15 19:18:49 +02:00
Robert Habermeier
25f1f22c86 use account_pair in feed 2016-06-15 18:41:02 +02:00
Robert Habermeier
0e3a15cadb add a state rebuilder 2016-06-15 17:46:40 +02:00
Robert Habermeier
d7498c1dd5 rename AccountReader to Account, give a separate module 2016-06-15 16:42:49 +02:00
Robert Habermeier
a2bb3f2832 account storage decoding implemented 2016-06-15 16:34:11 +02:00
Robert Habermeier
d26e038dc7 fix bug in account code storage, add to_thin_rlp for AccountReader 2016-06-15 15:59:21 +02:00
Robert Habermeier
9badb310ae remove snappy_buffer_size constant in favor of max_compressed_len 2016-06-15 13:05:00 +02:00
Robert Habermeier
53b3a6f0a1 use UntrustedRlp in to_block 2016-06-15 12:45:04 +02:00
Robert Habermeier
17a60920cb adjust comments 2016-06-14 19:14:41 +02:00
Robert Habermeier
3290f393bd block rlp compression 2016-06-14 18:34:27 +02:00
Robert Habermeier
8ed2b9c7d9 refactor out common parts of write_chunk 2016-06-14 15:32:49 +02:00
Robert Habermeier
dfb603dd08 compress into reusable buffers 2016-06-14 13:22:15 +02:00
Robert Habermeier
16e58958c9 use snappy compression on buffers 2016-06-14 12:32:51 +02:00
Robert Habermeier
b88eef5374 use append_empty_data rather than fibbing it 2016-06-14 11:48:34 +02:00
Robert Habermeier
835b158069 store account code if it exists 2016-06-13 21:05:06 +02:00
Robert Habermeier
5cc6a681c9 push pair onto block chunker buffer 2016-06-13 20:48:12 +02:00
Robert Habermeier
c36f47bbbf rename pv64_snapshot trace target to snapshot 2016-06-13 20:00:47 +02:00
Robert Habermeier
22a19819c6 rename pv64 module to snapshot 2016-06-13 20:00:00 +02:00