Commit Graph

236 Commits

Author SHA1 Message Date
Tomasz Drwięga
e4f0c0b215 Single DB (#1741)
* Consolidation migration

* Started db amalgamation

* Using client constants for columns

* Adding with_columns constructor

* Migrating to single db

* Fixing tests.

* test.sh without verbose

* Fixing warnings

* add migration tests that catch the bug

* make multiple migrations more robust

* add moved v9

* Merge branch 'noop-migrations' into single-db

* spurious line

* clean up migrations ordering

* update comment [ci skip]

* Bumping default number of max_open_files & re-ordering columns.

* fix merge

* fix ignored analysis tests

* Caching best block content

* Faster best_block_header

* Adding progress to v8 migration

* clean up warnings

* Separate hashes and bodies in the DB

* Separate hashes and bodies in the DB

* Fixed tests
2016-07-28 23:46:24 +02:00
keorn
02cf48681d Blocks and snapshot compression (#1687)
* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* new Compressible rlp trait

* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* DecompressingDecoder test

* initial compressing HashDB wrapper

* remove unused test

* change CompressedDB to struct wrapper with overlay

* simplify compressor

* failed RefCell attempt

* use denote to return reference

* compiled compresseddb

* compressdb test, add overlay emplace

* fix overlay reference count handling

* add immutable compresseddb, make account use hashdb

* simplify using trait objects

* enable hashdb for account

* initial state compression attempt

* wrap state db

* add tests for analyzing db

* add account predicate

* try to compress data fields as rlp too

* remove compression for storage trie

* add a compressing migration

* more compression stats tests

* fix migration import

* nested encoding compression test

* fix decompression, move db stats tests to rlpcompression

* added malformed rlp tests, cover a few edge cases

* new CompressingEncoder struct

* extend migrations to state

* first version working on the whole db

* clean up Compressible impl

* tests cleanup

* add a testing migration

* refactor deep compression using option, add simple compression

* put tests in a module

* fix compressed overlay loading

* simple compression for snapshots

* remove unused DecompressingDecoder

* add a general compressing migration

* add more common rlps to compress

* use static slices for swapper

* add precomputed hashes and invalid rlps

* make decoder private again

* cover more cases with tests

* style

* fix weird indentation

* remove possible panic in payload_info

* make prefix checking safe

* fix db existence check

* remove db dir from test

* pass usize by value [ci skip]

* Improve comment on panic removal.

* add common blocks db rlps

* add compression to blockchain db

* add blocks db migration

* fix the migrations

* remove state compression

* add a separate snapshot swapper

* ability to use different swappers and traversal

* update tests to new interface

* clean up code ordering

* update usage

* fix compilation

* remove unnecessary changes

* move methods to functions to reduce interface

* move test to module

* update common rlps to blocks db

* move tests to tests modules

* remove redundant &
2016-07-27 17:11:41 +02:00
Tomasz Drwięga
038862fa9d Fixing clippy warnings (#1660) 2016-07-19 09:25:51 +02:00
Robert Habermeier
4269867ca4 remove unnecessary assertion 2016-07-11 19:56:27 +02:00
Robert Habermeier
673cde6c7f add block tests 2016-07-11 18:31:18 +02:00
Robert Habermeier
49ba117f13 add account tests 2016-07-11 16:54:50 +02:00
Robert Habermeier
24ec696062 rebuild uncles hash from uncle headers 2016-07-11 15:19:01 +02:00
Robert Habermeier
da9e9692dc remove outdated comment; prove panickers; update manifest 2016-07-11 13:17:26 +02:00
Robert Habermeier
c7af950992 ignore warnings temporarily 2016-07-11 11:10:08 +02:00
Robert Habermeier
36dfa4743e merge accountdb migration 2016-07-09 17:33:14 +02:00
Robert Habermeier
456619001a parallelize account trie creation 2016-06-30 20:43:54 +02:00
Robert Habermeier
6ecd6eaa12 batch state trie writes 2016-06-29 18:37:17 +02:00
Robert Habermeier
49024a4f28 Merge with master 2016-06-29 14:46:29 +02:00
Robert Habermeier
3850ee64bb have StateRebuilder take a JournalDB and commit post-chunk 2016-06-17 12:56:57 +02:00
Robert Habermeier
f3c1643090 encode the account storage rlp with "append_raw" 2016-06-16 19:47:05 +02:00
Robert Habermeier
e4867d7cb9 properly rebuild state trie 2016-06-16 18:30:18 +02:00
Robert Habermeier
f45254d485 create chunks using append_raw 2016-06-16 16:39:42 +02:00
Robert Habermeier
0e917b9fa3 use ZST error type for snappy, use new snappy compression methods 2016-06-16 15:35:49 +02:00
Robert Habermeier
7ca5c115d8 validate all compressed buffers 2016-06-15 19:18:49 +02:00
Robert Habermeier
25f1f22c86 use account_pair in feed 2016-06-15 18:41:02 +02:00
Robert Habermeier
0e3a15cadb add a state rebuilder 2016-06-15 17:46:40 +02:00
Robert Habermeier
d7498c1dd5 rename AccountReader to Account, give a separate module 2016-06-15 16:42:49 +02:00
Robert Habermeier
a2bb3f2832 account storage decoding implemented 2016-06-15 16:34:11 +02:00
Robert Habermeier
d26e038dc7 fix bug in account code storage, add to_thin_rlp for AccountReader 2016-06-15 15:59:21 +02:00
Robert Habermeier
9badb310ae remove snappy_buffer_size constant in favor of max_compressed_len 2016-06-15 13:05:00 +02:00
Robert Habermeier
53b3a6f0a1 use UntrustedRlp in to_block 2016-06-15 12:45:04 +02:00
Robert Habermeier
17a60920cb adjust comments 2016-06-14 19:14:41 +02:00
Robert Habermeier
3290f393bd block rlp compression 2016-06-14 18:34:27 +02:00
Robert Habermeier
8ed2b9c7d9 refactor out common parts of write_chunk 2016-06-14 15:32:49 +02:00
Robert Habermeier
dfb603dd08 compress into reusable buffers 2016-06-14 13:22:15 +02:00
Robert Habermeier
16e58958c9 use snappy compression on buffers 2016-06-14 12:32:51 +02:00
Robert Habermeier
b88eef5374 use append_empty_data rather than fibbing it 2016-06-14 11:48:34 +02:00
Robert Habermeier
835b158069 store account code if it exists 2016-06-13 21:05:06 +02:00
Robert Habermeier
5cc6a681c9 push pair onto block chunker buffer 2016-06-13 20:48:12 +02:00
Robert Habermeier
c36f47bbbf rename pv64_snapshot trace target to snapshot 2016-06-13 20:00:47 +02:00
Robert Habermeier
22a19819c6 rename pv64 module to snapshot 2016-06-13 20:00:00 +02:00