Commit Graph

86 Commits

Author SHA1 Message Date
Arkadiy Paronyan
487dfb0208 Snapshot sync part 2 (#2098)
* Split block downloader into a module

* Snapshot sync progress

* Warp sync CLI option

* Increased snapshot chunk and ping timeouts

* Fixed an issue with delayed writes

* Updated bootnodes

* Don't run pending IO tasks on shutdown

* Optional first_block; removed insert_snapshot_block

* Fixing expect calls

* Fixed stalled sync

* style and docs

* Update block_sync.rs

[ci:skip]
2016-10-18 18:16:00 +02:00
Robert Habermeier
8263bd4be2 snapshot: single byte for empty accounts (#2625) 2016-10-14 15:45:45 +03:00
Robert Habermeier
f28b8352c1 Configurable history size in master (#2606)
* split journaldb commit into two functions: journal_under and mark_canonical

* use new commit mechanism in client

* Configurable history size in master

* Reduce DB history

* Configurable history size

* Set min history size

* Test

* Fixed a test and reduced the limit
2016-10-14 15:44:56 +03:00
Nikolay Volf
7526b1d44b Derive IPC interface only when ipc feature is on (#2463)
* derive -> ipc

* accident repair

* conditional ipc deriving

* fix test
2016-10-05 19:42:57 +02:00
Nikolay Volf
2b147616fd derive -> ipc 2016-10-04 19:20:16 +03:00
Nikolay Volf
e1d3b3fff8 Accounts bloom in master (#2426)
* bloom crate link

* database layout and outdated tests

* state db alterations

* v10 migration run

* using arc

* bloom migration

* migration fixes and mess

* fix tests
2016-10-03 12:02:43 +02:00
Arkadiy Paronyan
ad63780b4d Canonical state cache (master) (#2311)
* State cache

* Reduced copying data between caches

Whitespace and optional symbols

* Reduced copying data between caches

Whitespace and optional symbols

* Set a limit on storage cache

* Style and docs
2016-09-27 18:02:11 +02:00
Robert Habermeier
723d837d05 Snapshot format changes (#2234)
Closes #2213
Omit transaction and receipt roots from abridged block.
No longer use RLP compression.

Make ordered_trie_root generic over an iterator to save an allocation.

Breaks snapshot format backwards compatibility (with other 1.4 snapshots -- it's already been broken with 1.3).

Documentation will need updating
2016-09-22 19:47:03 +02:00
Robert Habermeier
a100b9d09e Remove panickers from trie iterators (#2209)
* port trie iterators to use error handling

* use new trie iterators in snapshot

allows proper recovery from a premature periodic snapshot
2016-09-21 12:56:13 +02:00
Tomasz Drwięga
7f7e74869c Bump clippy & Fix warnings (#2109)
* Bump clippy

* Fixing warnings
2016-09-16 23:03:26 +02:00
Robert Habermeier
83ddce011d ensure the target folder doesn't exist before renaming (#2074) 2016-09-13 10:33:03 +02:00
Robert Habermeier
edcc4080d5 fix failing master build: update tests to new init_restore signature. (#2069) 2016-09-11 20:17:05 +02:00
Robert Habermeier
dcfd7eab6d Local snapshot restore (#2058)
* restore from local snapshot

* update status with chunks done

* rework local restore trigger
2016-09-11 14:05:59 +02:00
Robert Habermeier
ff17174cf1 also test abort_restore 2016-09-08 12:44:10 +02:00
Robert Habermeier
756b7a3e67 test folder-deleting guards 2016-09-08 12:27:13 +02:00
Robert Habermeier
c65a5c8e9c add restored_is_equivalent test 2016-09-08 12:14:53 +02:00
Robert Habermeier
b0f6bf2e78 remove client_db field from snapshot service 2016-09-07 19:38:59 +02:00
Robert Habermeier
57d5c35bb6 Use proper database configuration in snapshots. (#2052)
* use proper database config in snapshot service

* add snapshot path to parity directories struct

* fix RPC tests
2016-09-07 15:27:28 +02:00
Robert Habermeier
541b14a4ab periodic snapshot tweaks (#2054)
* periodic snapshot tweaks

* set SNAPSHOT_HISTORY to 500
2016-09-07 15:27:14 +02:00
Robert Habermeier
f054a7b8d5 more info on current periodic snapshot 2016-09-06 17:44:11 +02:00
Robert Habermeier
46581e173d check block queue size before taking periodic snapshot 2016-09-06 15:49:44 +02:00
Robert Habermeier
1e2cdedc9b merge with master 2016-09-06 15:41:56 +02:00
Arkadiy Paronyan
5c5d9c8ccd Snapshot sync (#2047)
* PV64 sync

* Tests

* Client DB restore

* Snapshot restoration over IPC

* Upating test

* Minor tweaks

* Upating test
2016-09-06 15:31:13 +02:00
Robert Habermeier
46b1224f7c Merge branch 'master' into periodic_snapshot 2016-09-05 14:50:43 +02:00
Robert Habermeier
f0ef5e6943 keep snapshot watcher alive 2016-09-05 14:25:56 +02:00
Robert Habermeier
4e466f09db Add a few small snapshot tests (#2038)
* add manifest data test

* begin tests for io

* add unit tests for snapshot io

* service smoke test

* remove empty service module
2016-09-05 12:24:03 +02:00
Robert Habermeier
2bf235e226 use more mocking in tests 2016-09-05 12:17:21 +02:00
Robert Habermeier
d9eb87cae7 add guard for temporary directories 2016-09-02 19:00:20 +02:00
Robert Habermeier
e3749b3bc4 implement snapshot watcher 2016-09-02 18:28:47 +02:00
Robert Habermeier
04dee54cb3 add take_snapshot to snapshot service 2016-09-02 16:15:25 +02:00
Robert Habermeier
5dd56aa070 port ethcore 2016-09-01 14:29:59 +02:00
Nipunn Koorapati
4389742ca3 Make the block header struct's internals private (#2000)
* Make the block header struct's internals private

Currently, this involves a lot of explicit cloning, but we
could migrate the return types of the get_* functions to
be copies rather than references since they are mostly copy
types anyway.

I opted to eliminate the constructor in favor of using
Default::default() plus calling a bunch of setters. This
is similar to the model that a Google Protobuf client uses
and I think it looks fine.

* Drop some unnecessary cloning by comparing references

* Fix compiler errors from callsites in tests.
2016-08-29 11:35:23 +02:00
Robert Habermeier
1c19a807d9 Take control of recovered snapshots, start restoration asynchronously (#2010)
* take control of given snapshot

* start snapshot restoration asynchronously,
2016-08-25 22:20:44 +02:00
Robert Habermeier
2aef81cf90 remove internal locking from DBTransaction (#2003) 2016-08-25 16:43:56 +02:00
Robert Habermeier
b18407b9e3 Snapshot optimizations (#1991)
* apply RLP compression to abridged blocks

* add memorydb consolidate

* code hash optimization

* add warning to snapshot restoration CLI
2016-08-25 14:28:45 +02:00
Robert Habermeier
190e4db266 Trie query recording and AccountDB factory for no mangling (#1944)
* optionally use no mangling for accountdb

* add the recorder module

* get_recorded for tries, no virtual dispatch on readonly tries

* add recording test
2016-08-24 16:53:36 +02:00
Nikolay Volf
d7c184bd72 Merge pull request #1953 from ethcore/state_module
introduce ethcore/state module
2016-08-18 21:56:08 +04:00
Robert Habermeier
9adf77581c move column family constants into db module (#1955) 2016-08-18 18:24:49 +02:00
Robert Habermeier
eed7d006f1 Fix up tests, remove dead state-diff code 2016-08-18 18:13:32 +02:00
Marek Kotewicz
c39761c042 ethkey and ethstore use hash structures from bigint (#1851)
* Address renamed to H160 at bigint library level

* moved uint specific test from util to bigint library

* naming

* unifing hashes in progress

* unifing hashes

* cleanup redundant unwraps in tests

* fixed compiling
2016-08-15 15:09:00 +02:00
Tomasz Drwięga
a427208f79 Purging .derefs, fixing clippy warnings. (#1890)
* Fixing clippy warnings

* Purging derefs

* Simplifying engine derefs

* Simplifying more engine derefs
2016-08-10 16:29:40 +02:00
debris
974d537849 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-09 09:58:18 +02:00
Robert Habermeier
2f1ade8116 take snapshot at specified block and slightly better informants (#1873)
* prettier informant for snapshot creation

* allow taking snapshot at a given block

* minor tweaks

* elaborate on cli
2016-08-08 18:41:30 +02:00
debris
88c5f555a9 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-06 00:03:07 +02:00
Robert Habermeier
e72fc5398a miner and client take spec reference (#1853)
* miner and client take spec reference

* fix tests
2016-08-05 23:33:55 +02:00
Robert Habermeier
76a7246369 Snapshot creation and restoration (#1679)
* to_rlp takes self by-reference

* clean up some derefs

* out-of-order insertion for blockchain

* implement block rebuilder without verification

* group block chunk header into struct

* block rebuilder does verification

* integrate snapshot service with client service; flesh out implementation more

* initial implementation of snapshot service

* remove snapshottaker trait

* snapshot writer trait with packed and loose implementations

* write chunks using "snapshotwriter" in service

* have snapshot taking use snapshotwriter

* implement snapshot readers

* back up client dbs when replacing

* use snapshot reader in snapshot service

* describe offset format

* use new get_db_path in parity, allow some errors in service

* blockchain formatting

* implement parity snapshot

* implement snapshot restore

* force blocks to be submitted in order

* fix bug loading block hashes in packed reader

* fix seal field loading

* fix uncle hash computation

* fix a few bugs

* store genesis state in db. reverse block chunk order in packed writer

* allow out-of-order import for blocks

* bring restoration types together

* only snapshot the last 30000 blocks

* restore into overlaydb instead of journaldb

* commit version to database

* use memorydbs and commit directly

* fix trie test compilation

* fix failing tests

* sha3_null_rlp, not H256::zero

* move overlaydb to ref_overlaydb, add new overlaydb without on-disk rc

* port archivedb to new overlaydb

* add deletion mode tests for overlaydb

* use new overlaydb, check state root at end

* share chain info between state and block snapshotting

* create blocks snapshot using blockchain directly

* allow snapshot from arbitrary block, remove panickers from snapshot creation

* begin test framework

* blockchain chunking test

* implement stateproducer::tick

* state snapshot test

* create block and state chunks concurrently, better restoration informant

* fix tests

* add deletion mode tests for overlaydb

* address comments

* more tests

* Fix up tests.

* remove a few printlns

* add a little more documentation to `commit`

* fix tests

* fix ref_overlaydb test names

* snapshot command skeleton

* revert ref_overlaydb renaming

* reimplement snapshot commands

* fix many errors

* everything but inject

* get ethcore compiling

* get snapshot tests passing again

* instrument snapshot commands again

* fix fallout from other changes, mark snapshots as experimental

* optimize injection patterns

* do two injections

* fix up tests

* take snapshots from 1000 blocks efore

* address minor comments

* fix a few io crate related errors

* clarify names about total difficulty

[ci skip]
2016-08-05 17:00:46 +02:00
debris
03c3d16744 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-04 08:41:30 +02:00
Tomasz Drwięga
deceb5fd56 Bumping clippy & fixing warnings (#1823)
* Bumping clippy

* Fixing clippy warnings

* Cargo.lock (latest nightly?)
2016-08-03 19:01:48 +02:00
Robert Habermeier
11b65ce53d Remove (almost all) panickers from trie module (#1776)
* memorydb ub patch and other cleanup

* fix denote invocations

* move trie traits into trie module

* replace "denote" with shim

* triedb returns results and no longer panics

* fix warnings

* get ethcore compiling

* warn on trie errors in ethcore

* remove unsafety from node decoder

* restore broken denote behavior for this branch

* fix overlayrecent fallout

* fix triedb tests

* remove unwrap in state

* alter Trie::get to return Result<Option<_>>

* fix refcell error in require

* fix test warnings

* fix json tests

* whitespace

[ci:skip]

* Avoid unneeded match/indentation

* whitespace

* prettify map_or_else

* remove test warning
2016-08-03 18:35:48 +02:00
debris
e8c451ac82 cleaning up hash reexports 2016-08-03 18:05:17 +02:00