Commit Graph

248 Commits

Author SHA1 Message Date
Robert Habermeier d069b98b45 PoA: Wait for transition finality before applying (#5774)
* final engine changes

* migration to v13

* adding and removing pending transitions

* epoch_transition_for

* port snapshot to new engine methods

* final validator set interface

* fix compiler errors

* revert v13/epoch_depth transition

* make call on new epoch

* rolling finality checker

* tests for finality checker

* constructing finality proof upon pending transition

* fix warnings and finality proof checking

* fix compiler warnings in tests

* test fixes

* don't include genesis in finality checking

* change snapshot test chain building logic

* minor refactorings

* fetch epoch transition based on parent, fix divide-by-zero in SimpleList

* fix formatting

* fix ABIs and finality checking in snapshot restoration

* encode signal number in proof

* create more blocks at the end of tests

* update gist to accurate contract code

* test for epoch_transition_for

* fix tests with immediateTransitions parameter

* disable force flag after forcing

* rename ValidatorsChanged to InitiateChange and finalizeSignal to finalizeChange

* a few more validator set tests
2017-06-28 13:17:36 +02:00
Arkadiy Paronyan 57626b60e7 EIP-168, 169: Dust protection (#4757)
* Dust protection

* Track touched accounts in the substate

* Minor alterations
2017-06-28 09:10:57 +02:00
Arkadiy Paronyan e6a31e7543 EIP-210 BLOCKHASH changes (#5505)
* EIP-210

* comment
2017-05-30 11:52:33 +02:00
Nikolay Volf e1fef5c732 Update dependencies and bigint api (#5685)
* update to latest bigint

* bump elastic array and deps

* fix rlp tests

* also update all smallvec deps

* fix doc test

* reduce parking in attempt to fix CI bug

* fix from/into electum bug

* remove duplicate imports
2017-05-24 12:31:33 +02:00
Tomasz Drwięga f47cbe0be6 Adding CLI options: port shift and unsafe expose. (#5677)
* Adding CLI option for port shift and unsafe expose.

* Fixing IPC path.

* Fix hosts when attempting to expose on all interfaces.

* Fixing test.

* Fix typo.
2017-05-23 12:24:32 +02:00
Robert Habermeier 4c5e4ac8da PoA warp implementation (#5488)
* separate modules for consensus snapshot chunks

* bulk of authority warp logic

* finish authority warp implementation

* squash warnings and enable authority snapshot mode

* test harness for PoA

* fiddle with harness

* epoch generation proof fixes

* update constructor code

* store epoch transition proof after block commit

* basic snap and restore test

* use keyvaluedb in state restoration

* decompress chunks

* fix encoding issues

* fixed-to-contract-to-contract test

* implement ancient block import

* restore genesis transition in PoW snapshot

* add format version method to snapshot components

* supported version numbers in snapshot_components

* allow returning of ancient epoch transitions

* genesis hash mismatch check

* remove commented code
2017-05-17 12:41:33 +02:00
Robert Habermeier 6a5702f27c address grumbles 2017-04-24 13:14:50 +02:00
Robert Habermeier 2ec3397b7d snapshot chunk and restore traits 2017-04-19 20:31:53 +02:00
Robert Habermeier 2d8a8bd3e5 engine changes 2017-04-12 14:41:19 +02:00
Robert Habermeier a254b2098f more useful Engine::verify_seal 2017-04-11 17:07:04 +02:00
arkpar 03ec27be66 Removed assert 2017-04-05 16:50:06 +02:00
arkpar 81db3461fe Added an assert 2017-04-05 12:27:37 +02:00
arkpar b840ab8f8b Compact chunks 2017-03-31 16:31:12 +02:00
Tomasz Drwięga d2394d3ac3 Updating paths to repos. (#5330)
* Updating paths to repos.

* Updating rotor and libusb

* Fixing other occurrences
2017-03-29 15:17:27 +02:00
arkpar 387a5fb03b Doc 2017-03-26 13:26:20 +02:00
arkpar b0a9c1e0fa Limit by entry count 2017-03-25 15:25:20 +01:00
arkpar b52c7bba4b Fewer allocations 2017-03-25 13:22:25 +01:00
arkpar 4ef89b5ccb Fine grained snapshot chunking 2017-03-25 13:22:25 +01:00
Robert Habermeier 109012cae8 force earliest era set in snapshot restore (#5021) 2017-03-25 10:00:50 +01:00
Marek Kotewicz 044d070667 rlp deserialization refactor, 30% faster (#4901)
* fixed naming of rlp modules

* RlpStream cleanup

* appending short rlp lists (0...55 bytes) is 25% faster

* RlpStream does not use bytes module, nor trait Stream

* removed unused code from rlp module

* compiling ethcore-util with new rlp serialization

* compiling parity with new rlp serialization

* fixed compiling ethcore-light with new rlp serialization

* fixed compiling ethsync with new rlp serialization

* moved rlp benches and rlp tests

* rlp deserialization refactor, 30% faster

* removed redundant comment, print

* fixed compiling parity with new rlp deserialization

* removed redundant double-space

* fixed failing test

* updated rlp docs, removed unused traits

* fixed rlp benchmarks

* replace usage of WriteBytesExt with ByteOrder

* removed unused, commented out code

* fixed merge conflict
2017-03-22 14:41:46 +01:00
Marek Kotewicz a555686bcd rlp serialization refactor (#4873)
* fixed naming of rlp modules

* RlpStream cleanup

* appending short rlp lists (0...55 bytes) is 25% faster

* RlpStream does not use bytes module, nor trait Stream

* removed unused code from rlp module

* compiling ethcore-util with new rlp serialization

* compiling parity with new rlp serialization

* fixed compiling ethcore-light with new rlp serialization

* fixed compiling ethsync with new rlp serialization

* removed redundant comment, print

* removed redundant double-space

* replace usage of WriteBytesExt with ByteOrder
2017-03-20 19:14:29 +01:00
Marek Kotewicz 3fe3353696 removed redundant FixedHash trait, fixes #4029 (#4866) 2017-03-11 22:58:15 +04:00
Robert Habermeier 62b340f2b9 Save pending local transactions in the database (#4566)
* Create new column family for local node info

* remove DBTransaction::new reliance on DB

* KeyValueDB trait

* InMemory KeyValueDB implementation

* journaldb generic over KVDB

* make most of `ethcore` generic over KVDB

* fix json tests compilation

* get all tests compiling

* implement local store (just for transactions)

* finish local store API, test

* put everything into place

* better test for skipping bad transactions

* fix warning

* update local store every 15 minutes

* remove superfluous `{}`s
2017-02-20 17:21:55 +01:00
Gav Wood 8404edb656 Fix whitespace (#4299)
* Fix whitespace.

* Update copyright years/owner.

* Push release only for tags.
2017-01-25 18:51:41 +01:00
keorn b7f9b304c1 Chain scoring (#4218)
* chain scoring

* use current steps
2017-01-23 17:27:11 +03:00
Tomasz Drwięga e11353f94c UnverifiedTransaction type (#4134)
* Introducing ValidSignedTransaction

* Verifiying transactions in engines

* Widening use of VerifiedSignedTransaction

* Renaming Transactions

* Uncommenting banning queue & Fixing tests

* Fixing json tests

* Fixing pre-homestead test

* Fixing imports

* Addressing grumbles

* Fixing test
2017-01-13 08:51:36 +00:00
Robert Habermeier 5d2cf22ef4 use basic_account in snapshot 2017-01-03 17:05:27 +01:00
Robert Habermeier fe1f542c4f Owning views of blockchain data (#3982)
* owning views of blockchain data

* port blockchain and client traits to owning views

* fix ethcore tests

* use strong headers and bodies in ethcore_light

* port ethsync to use owning views

* port rpc to owning views

* port parity informant and blockchain export
2016-12-28 12:44:51 +00:00
Robert Habermeier 8125b5690c Port `try` macro to new `?` operator. (#3962)
* initial untry sweep

* restore try in ipc codegen, fix inference

* change a few missed try instances
2016-12-27 12:53:56 +01:00
keorn fa504e510f Merge branch 'master' into auth-bft 2016-12-12 21:28:46 +01:00
Gav Wood 17cc2be4b5
Rename ethcore -> parity in license. 2016-12-11 19:30:54 +01:00
keorn f3af0f46be Merge branch 'master' into auth-bft 2016-12-10 18:14:35 +01:00
Gav Wood 5f1fcf95e0
Make *ID names consistent with std Rust (Id) 2016-12-09 23:01:43 +01:00
keorn 3ebfa1481d better proposal block handling 2016-12-08 12:03:34 +01:00
keorn d9eb5e7f1d remove uncles 2016-12-07 09:32:36 +01:00
keorn 8f641e67f3 Merge branch 'engine-password' into auth-bft 2016-12-06 13:13:02 +00:00
keorn 53b479fb7a pass engine to snapshot service 2016-12-05 15:43:46 +00:00
keorn c61a0e97b3 make engine determine block order 2016-12-05 15:20:32 +00:00
keorn 155da50fcd Merge branch 'master' into auth-bft 2016-11-30 13:06:21 +00:00
Tomasz Drwięga 2b3d100de2 Fixing some clippy warnings 2016-11-28 15:54:22 +01:00
keorn 66526af5a8 pass engine in tests 2016-11-21 18:53:53 +00:00
keorn d19e8c5505 Merge remote-tracking branch 'parity/master' into bft 2016-11-15 10:26:37 +00:00
Robert Habermeier 18153b8d9e Merge branch 'master' into missing-code-by-prefix 2016-11-14 14:02:19 +01:00
Robert Habermeier 37f49aac1b abort snapshot restoration faster (#3356)
* abort snapshot restoration faster

* flag-checking tests
2016-11-13 13:52:53 +01:00
Robert Habermeier 87c668ba9c test previous code fetch 2016-11-11 18:24:12 +01:00
Gav Wood a7ad75d851 whitespace
[ci:skip]
2016-11-11 17:32:54 +01:00
Robert Habermeier ed135bb9dc fetch known code from the database during restoration
previously it kept all seen code in memory, leading to
high memory usage by the end of state restoration
2016-11-11 17:18:31 +01:00
keorn ea960f1c27 Merge remote-tracking branch 'parity/master' into bft 2016-11-07 11:34:45 +00:00
Arkadiy Paronyan bccc56b6b0 mio version bump (#2982) 2016-10-30 09:56:34 +01:00
Tomasz Drwięga f17cad9dff Fixing clippy warnings 2 (#2961) 2016-10-29 13:08:49 +02:00
Robert Habermeier 2806f1d4c9 Even more snapshot validity checks (#2935)
* clarify "cancelled periodic snapshot" message

* more rigorous checks for snapshot validity

* verify ancient blocks on import

* limit number of fed blocks

* make it possible to feed snapshot service canonical hashes

* fix failing test build

* swap ethash DAG only when more recent
2016-10-28 16:10:30 +02:00
Tomasz Drwięga 88997801d0 Clippy bump (#2877)
* Bumping clippy

* Fixing warnings

* Fix the "fix"
2016-10-27 08:28:12 +02:00
Arkadiy Paronyan f024acd329 More snapshot fixes and optimizations (#2883)
* More snapshot fixes and optimizations

* db drop
2016-10-26 16:14:13 +02:00
Arkadiy Paronyan 5b978be034 Get rid of MemoryDB denote (#2881) 2016-10-26 13:53:47 +02:00
Arkadiy Paronyan 135d5d0e4c Snapshot fixes and optimizations (#2863) 2016-10-25 18:40:01 +02:00
Robert Habermeier bc81ae0407 Snapshot and blockchain stability improvements (#2843)
* allow taking snapshot from just-restored database without error

* make creation informant less spammy

* Ancestry iterator failure-resilient

* make uncle hash searching resilient to incomplete chain

* deduce pre-chunk info from last written block's details
2016-10-24 18:27:23 +02:00
keorn 866ab9c7a3 Check queue to determine major importing (#2763)
* simplify major sync detection

* fix typos

* fix merge

* more realistic EthTester

* add new synced state

* remove Blocks synced state

* move is_major_importing to rpc crate and check queue

* add tests
2016-10-20 23:36:18 +02:00
Arkadiy Paronyan ae853a7557 Don't add empty accounts to bloom (#2753) 2016-10-20 16:49:27 +02:00
keorn aa52b04e31 Unify major syncing detection (#2699)
* simplify major sync detection

* fix typos

* fix merge

* more realistic EthTester

* add new synced state
2016-10-19 18:35:39 +02:00
Arkadiy Paronyan 487dfb0208 Snapshot sync part 2 (#2098)
* Split block downloader into a module

* Snapshot sync progress

* Warp sync CLI option

* Increased snapshot chunk and ping timeouts

* Fixed an issue with delayed writes

* Updated bootnodes

* Don't run pending IO tasks on shutdown

* Optional first_block; removed insert_snapshot_block

* Fixing expect calls

* Fixed stalled sync

* style and docs

* Update block_sync.rs

[ci:skip]
2016-10-18 18:16:00 +02:00
Robert Habermeier 8263bd4be2 snapshot: single byte for empty accounts (#2625) 2016-10-14 15:45:45 +03:00
Robert Habermeier f28b8352c1 Configurable history size in master (#2606)
* split journaldb commit into two functions: journal_under and mark_canonical

* use new commit mechanism in client

* Configurable history size in master

* Reduce DB history

* Configurable history size

* Set min history size

* Test

* Fixed a test and reduced the limit
2016-10-14 15:44:56 +03:00
keorn e343153f06 mixed merge and changes... 2016-10-11 18:37:31 +01:00
Nikolay Volf 7526b1d44b Derive IPC interface only when ipc feature is on (#2463)
* derive -> ipc

* accident repair

* conditional ipc deriving

* fix test
2016-10-05 19:42:57 +02:00
keorn 1f56588b87 Merge remote-tracking branch 'parity/master' into bft
Conflicts:
	ethcore/src/client/client.rs
2016-10-05 14:57:14 +01:00
keorn 67c24dcb95 use Engine to order blockchain 2016-10-05 14:29:35 +01:00
Nikolay Volf 2b147616fd derive -> ipc 2016-10-04 19:20:16 +03:00
Nikolay Volf e1d3b3fff8 Accounts bloom in master (#2426)
* bloom crate link

* database layout and outdated tests

* state db alterations

* v10 migration run

* using arc

* bloom migration

* migration fixes and mess

* fix tests
2016-10-03 12:02:43 +02:00
Arkadiy Paronyan ad63780b4d Canonical state cache (master) (#2311)
* State cache

* Reduced copying data between caches

Whitespace and optional symbols

* Reduced copying data between caches

Whitespace and optional symbols

* Set a limit on storage cache

* Style and docs
2016-09-27 18:02:11 +02:00
Robert Habermeier 723d837d05 Snapshot format changes (#2234)
Closes #2213
Omit transaction and receipt roots from abridged block.
No longer use RLP compression.

Make ordered_trie_root generic over an iterator to save an allocation.

Breaks snapshot format backwards compatibility (with other 1.4 snapshots -- it's already been broken with 1.3).

Documentation will need updating
2016-09-22 19:47:03 +02:00
Robert Habermeier a100b9d09e Remove panickers from trie iterators (#2209)
* port trie iterators to use error handling

* use new trie iterators in snapshot

allows proper recovery from a premature periodic snapshot
2016-09-21 12:56:13 +02:00
Tomasz Drwięga 7f7e74869c Bump clippy & Fix warnings (#2109)
* Bump clippy

* Fixing warnings
2016-09-16 23:03:26 +02:00
Robert Habermeier 83ddce011d ensure the target folder doesn't exist before renaming (#2074) 2016-09-13 10:33:03 +02:00
Robert Habermeier edcc4080d5 fix failing master build: update tests to new init_restore signature. (#2069) 2016-09-11 20:17:05 +02:00
Robert Habermeier dcfd7eab6d Local snapshot restore (#2058)
* restore from local snapshot

* update status with chunks done

* rework local restore trigger
2016-09-11 14:05:59 +02:00
Robert Habermeier ff17174cf1 also test abort_restore 2016-09-08 12:44:10 +02:00
Robert Habermeier 756b7a3e67 test folder-deleting guards 2016-09-08 12:27:13 +02:00
Robert Habermeier c65a5c8e9c add restored_is_equivalent test 2016-09-08 12:14:53 +02:00
Robert Habermeier b0f6bf2e78 remove client_db field from snapshot service 2016-09-07 19:38:59 +02:00
Robert Habermeier 57d5c35bb6 Use proper database configuration in snapshots. (#2052)
* use proper database config in snapshot service

* add snapshot path to parity directories struct

* fix RPC tests
2016-09-07 15:27:28 +02:00
Robert Habermeier 541b14a4ab periodic snapshot tweaks (#2054)
* periodic snapshot tweaks

* set SNAPSHOT_HISTORY to 500
2016-09-07 15:27:14 +02:00
Robert Habermeier f054a7b8d5 more info on current periodic snapshot 2016-09-06 17:44:11 +02:00
Robert Habermeier 46581e173d check block queue size before taking periodic snapshot 2016-09-06 15:49:44 +02:00
Robert Habermeier 1e2cdedc9b merge with master 2016-09-06 15:41:56 +02:00
Arkadiy Paronyan 5c5d9c8ccd Snapshot sync (#2047)
* PV64 sync

* Tests

* Client DB restore

* Snapshot restoration over IPC

* Upating test

* Minor tweaks

* Upating test
2016-09-06 15:31:13 +02:00
Robert Habermeier 46b1224f7c Merge branch 'master' into periodic_snapshot 2016-09-05 14:50:43 +02:00
Robert Habermeier f0ef5e6943 keep snapshot watcher alive 2016-09-05 14:25:56 +02:00
Robert Habermeier 4e466f09db Add a few small snapshot tests (#2038)
* add manifest data test

* begin tests for io

* add unit tests for snapshot io

* service smoke test

* remove empty service module
2016-09-05 12:24:03 +02:00
Robert Habermeier 2bf235e226 use more mocking in tests 2016-09-05 12:17:21 +02:00
Robert Habermeier d9eb87cae7 add guard for temporary directories 2016-09-02 19:00:20 +02:00
Robert Habermeier e3749b3bc4 implement snapshot watcher 2016-09-02 18:28:47 +02:00
Robert Habermeier 04dee54cb3 add take_snapshot to snapshot service 2016-09-02 16:15:25 +02:00
Robert Habermeier 5dd56aa070 port ethcore 2016-09-01 14:29:59 +02:00
Nipunn Koorapati 4389742ca3 Make the block header struct's internals private (#2000)
* Make the block header struct's internals private

Currently, this involves a lot of explicit cloning, but we
could migrate the return types of the get_* functions to
be copies rather than references since they are mostly copy
types anyway.

I opted to eliminate the constructor in favor of using
Default::default() plus calling a bunch of setters. This
is similar to the model that a Google Protobuf client uses
and I think it looks fine.

* Drop some unnecessary cloning by comparing references

* Fix compiler errors from callsites in tests.
2016-08-29 11:35:23 +02:00
Robert Habermeier 1c19a807d9 Take control of recovered snapshots, start restoration asynchronously (#2010)
* take control of given snapshot

* start snapshot restoration asynchronously,
2016-08-25 22:20:44 +02:00
Robert Habermeier 2aef81cf90 remove internal locking from DBTransaction (#2003) 2016-08-25 16:43:56 +02:00
Robert Habermeier b18407b9e3 Snapshot optimizations (#1991)
* apply RLP compression to abridged blocks

* add memorydb consolidate

* code hash optimization

* add warning to snapshot restoration CLI
2016-08-25 14:28:45 +02:00
Robert Habermeier 190e4db266 Trie query recording and AccountDB factory for no mangling (#1944)
* optionally use no mangling for accountdb

* add the recorder module

* get_recorded for tries, no virtual dispatch on readonly tries

* add recording test
2016-08-24 16:53:36 +02:00
Nikolay Volf d7c184bd72 Merge pull request #1953 from ethcore/state_module
introduce ethcore/state module
2016-08-18 21:56:08 +04:00
Robert Habermeier 9adf77581c move column family constants into db module (#1955) 2016-08-18 18:24:49 +02:00
Robert Habermeier eed7d006f1 Fix up tests, remove dead state-diff code 2016-08-18 18:13:32 +02:00
Marek Kotewicz c39761c042 ethkey and ethstore use hash structures from bigint (#1851)
* Address renamed to H160 at bigint library level

* moved uint specific test from util to bigint library

* naming

* unifing hashes in progress

* unifing hashes

* cleanup redundant unwraps in tests

* fixed compiling
2016-08-15 15:09:00 +02:00
Tomasz Drwięga a427208f79 Purging .derefs, fixing clippy warnings. (#1890)
* Fixing clippy warnings

* Purging derefs

* Simplifying engine derefs

* Simplifying more engine derefs
2016-08-10 16:29:40 +02:00
debris 974d537849 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-09 09:58:18 +02:00
Robert Habermeier 2f1ade8116 take snapshot at specified block and slightly better informants (#1873)
* prettier informant for snapshot creation

* allow taking snapshot at a given block

* minor tweaks

* elaborate on cli
2016-08-08 18:41:30 +02:00
debris 88c5f555a9 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-06 00:03:07 +02:00
Robert Habermeier e72fc5398a miner and client take spec reference (#1853)
* miner and client take spec reference

* fix tests
2016-08-05 23:33:55 +02:00
Robert Habermeier 76a7246369 Snapshot creation and restoration (#1679)
* to_rlp takes self by-reference

* clean up some derefs

* out-of-order insertion for blockchain

* implement block rebuilder without verification

* group block chunk header into struct

* block rebuilder does verification

* integrate snapshot service with client service; flesh out implementation more

* initial implementation of snapshot service

* remove snapshottaker trait

* snapshot writer trait with packed and loose implementations

* write chunks using "snapshotwriter" in service

* have snapshot taking use snapshotwriter

* implement snapshot readers

* back up client dbs when replacing

* use snapshot reader in snapshot service

* describe offset format

* use new get_db_path in parity, allow some errors in service

* blockchain formatting

* implement parity snapshot

* implement snapshot restore

* force blocks to be submitted in order

* fix bug loading block hashes in packed reader

* fix seal field loading

* fix uncle hash computation

* fix a few bugs

* store genesis state in db. reverse block chunk order in packed writer

* allow out-of-order import for blocks

* bring restoration types together

* only snapshot the last 30000 blocks

* restore into overlaydb instead of journaldb

* commit version to database

* use memorydbs and commit directly

* fix trie test compilation

* fix failing tests

* sha3_null_rlp, not H256::zero

* move overlaydb to ref_overlaydb, add new overlaydb without on-disk rc

* port archivedb to new overlaydb

* add deletion mode tests for overlaydb

* use new overlaydb, check state root at end

* share chain info between state and block snapshotting

* create blocks snapshot using blockchain directly

* allow snapshot from arbitrary block, remove panickers from snapshot creation

* begin test framework

* blockchain chunking test

* implement stateproducer::tick

* state snapshot test

* create block and state chunks concurrently, better restoration informant

* fix tests

* add deletion mode tests for overlaydb

* address comments

* more tests

* Fix up tests.

* remove a few printlns

* add a little more documentation to `commit`

* fix tests

* fix ref_overlaydb test names

* snapshot command skeleton

* revert ref_overlaydb renaming

* reimplement snapshot commands

* fix many errors

* everything but inject

* get ethcore compiling

* get snapshot tests passing again

* instrument snapshot commands again

* fix fallout from other changes, mark snapshots as experimental

* optimize injection patterns

* do two injections

* fix up tests

* take snapshots from 1000 blocks efore

* address minor comments

* fix a few io crate related errors

* clarify names about total difficulty

[ci skip]
2016-08-05 17:00:46 +02:00
debris 03c3d16744 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-04 08:41:30 +02:00
Tomasz Drwięga deceb5fd56 Bumping clippy & fixing warnings (#1823)
* Bumping clippy

* Fixing clippy warnings

* Cargo.lock (latest nightly?)
2016-08-03 19:01:48 +02:00
Robert Habermeier 11b65ce53d Remove (almost all) panickers from trie module (#1776)
* memorydb ub patch and other cleanup

* fix denote invocations

* move trie traits into trie module

* replace "denote" with shim

* triedb returns results and no longer panics

* fix warnings

* get ethcore compiling

* warn on trie errors in ethcore

* remove unsafety from node decoder

* restore broken denote behavior for this branch

* fix overlayrecent fallout

* fix triedb tests

* remove unwrap in state

* alter Trie::get to return Result<Option<_>>

* fix refcell error in require

* fix test warnings

* fix json tests

* whitespace

[ci:skip]

* Avoid unneeded match/indentation

* whitespace

* prettify map_or_else

* remove test warning
2016-08-03 18:35:48 +02:00
debris e8c451ac82 cleaning up hash reexports 2016-08-03 18:05:17 +02:00
Tomasz Drwięga e4f0c0b215 Single DB (#1741)
* Consolidation migration

* Started db amalgamation

* Using client constants for columns

* Adding with_columns constructor

* Migrating to single db

* Fixing tests.

* test.sh without verbose

* Fixing warnings

* add migration tests that catch the bug

* make multiple migrations more robust

* add moved v9

* Merge branch 'noop-migrations' into single-db

* spurious line

* clean up migrations ordering

* update comment [ci skip]

* Bumping default number of max_open_files & re-ordering columns.

* fix merge

* fix ignored analysis tests

* Caching best block content

* Faster best_block_header

* Adding progress to v8 migration

* clean up warnings

* Separate hashes and bodies in the DB

* Separate hashes and bodies in the DB

* Fixed tests
2016-07-28 23:46:24 +02:00
keorn 02cf48681d Blocks and snapshot compression (#1687)
* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* new Compressible rlp trait

* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* DecompressingDecoder test

* initial compressing HashDB wrapper

* remove unused test

* change CompressedDB to struct wrapper with overlay

* simplify compressor

* failed RefCell attempt

* use denote to return reference

* compiled compresseddb

* compressdb test, add overlay emplace

* fix overlay reference count handling

* add immutable compresseddb, make account use hashdb

* simplify using trait objects

* enable hashdb for account

* initial state compression attempt

* wrap state db

* add tests for analyzing db

* add account predicate

* try to compress data fields as rlp too

* remove compression for storage trie

* add a compressing migration

* more compression stats tests

* fix migration import

* nested encoding compression test

* fix decompression, move db stats tests to rlpcompression

* added malformed rlp tests, cover a few edge cases

* new CompressingEncoder struct

* extend migrations to state

* first version working on the whole db

* clean up Compressible impl

* tests cleanup

* add a testing migration

* refactor deep compression using option, add simple compression

* put tests in a module

* fix compressed overlay loading

* simple compression for snapshots

* remove unused DecompressingDecoder

* add a general compressing migration

* add more common rlps to compress

* use static slices for swapper

* add precomputed hashes and invalid rlps

* make decoder private again

* cover more cases with tests

* style

* fix weird indentation

* remove possible panic in payload_info

* make prefix checking safe

* fix db existence check

* remove db dir from test

* pass usize by value [ci skip]

* Improve comment on panic removal.

* add common blocks db rlps

* add compression to blockchain db

* add blocks db migration

* fix the migrations

* remove state compression

* add a separate snapshot swapper

* ability to use different swappers and traversal

* update tests to new interface

* clean up code ordering

* update usage

* fix compilation

* remove unnecessary changes

* move methods to functions to reduce interface

* move test to module

* update common rlps to blocks db

* move tests to tests modules

* remove redundant &
2016-07-27 17:11:41 +02:00
Tomasz Drwięga 038862fa9d Fixing clippy warnings (#1660) 2016-07-19 09:25:51 +02:00
Robert Habermeier 4269867ca4 remove unnecessary assertion 2016-07-11 19:56:27 +02:00
Robert Habermeier 673cde6c7f add block tests 2016-07-11 18:31:18 +02:00
Robert Habermeier 49ba117f13 add account tests 2016-07-11 16:54:50 +02:00
Robert Habermeier 24ec696062 rebuild uncles hash from uncle headers 2016-07-11 15:19:01 +02:00
Robert Habermeier da9e9692dc remove outdated comment; prove panickers; update manifest 2016-07-11 13:17:26 +02:00
Robert Habermeier c7af950992 ignore warnings temporarily 2016-07-11 11:10:08 +02:00
Robert Habermeier 36dfa4743e merge accountdb migration 2016-07-09 17:33:14 +02:00
Robert Habermeier 456619001a parallelize account trie creation 2016-06-30 20:43:54 +02:00
Robert Habermeier 6ecd6eaa12 batch state trie writes 2016-06-29 18:37:17 +02:00
Robert Habermeier 49024a4f28 Merge with master 2016-06-29 14:46:29 +02:00
Robert Habermeier 3850ee64bb have StateRebuilder take a JournalDB and commit post-chunk 2016-06-17 12:56:57 +02:00
Robert Habermeier f3c1643090 encode the account storage rlp with "append_raw" 2016-06-16 19:47:05 +02:00
Robert Habermeier e4867d7cb9 properly rebuild state trie 2016-06-16 18:30:18 +02:00
Robert Habermeier f45254d485 create chunks using append_raw 2016-06-16 16:39:42 +02:00
Robert Habermeier 0e917b9fa3 use ZST error type for snappy, use new snappy compression methods 2016-06-16 15:35:49 +02:00
Robert Habermeier 7ca5c115d8 validate all compressed buffers 2016-06-15 19:18:49 +02:00
Robert Habermeier 25f1f22c86 use account_pair in feed 2016-06-15 18:41:02 +02:00
Robert Habermeier 0e3a15cadb add a state rebuilder 2016-06-15 17:46:40 +02:00
Robert Habermeier d7498c1dd5 rename AccountReader to Account, give a separate module 2016-06-15 16:42:49 +02:00
Robert Habermeier a2bb3f2832 account storage decoding implemented 2016-06-15 16:34:11 +02:00
Robert Habermeier d26e038dc7 fix bug in account code storage, add to_thin_rlp for AccountReader 2016-06-15 15:59:21 +02:00
Robert Habermeier 9badb310ae remove snappy_buffer_size constant in favor of max_compressed_len 2016-06-15 13:05:00 +02:00
Robert Habermeier 53b3a6f0a1 use UntrustedRlp in to_block 2016-06-15 12:45:04 +02:00
Robert Habermeier 17a60920cb adjust comments 2016-06-14 19:14:41 +02:00
Robert Habermeier 3290f393bd block rlp compression 2016-06-14 18:34:27 +02:00
Robert Habermeier 8ed2b9c7d9 refactor out common parts of write_chunk 2016-06-14 15:32:49 +02:00
Robert Habermeier dfb603dd08 compress into reusable buffers 2016-06-14 13:22:15 +02:00
Robert Habermeier 16e58958c9 use snappy compression on buffers 2016-06-14 12:32:51 +02:00
Robert Habermeier b88eef5374 use append_empty_data rather than fibbing it 2016-06-14 11:48:34 +02:00
Robert Habermeier 835b158069 store account code if it exists 2016-06-13 21:05:06 +02:00
Robert Habermeier 5cc6a681c9 push pair onto block chunker buffer 2016-06-13 20:48:12 +02:00
Robert Habermeier c36f47bbbf rename pv64_snapshot trace target to snapshot 2016-06-13 20:00:47 +02:00
Robert Habermeier 22a19819c6 rename pv64 module to snapshot 2016-06-13 20:00:00 +02:00