Commit Graph

39 Commits

Author SHA1 Message Date
Tomasz Drwięga
e4f0c0b215 Single DB (#1741)
* Consolidation migration

* Started db amalgamation

* Using client constants for columns

* Adding with_columns constructor

* Migrating to single db

* Fixing tests.

* test.sh without verbose

* Fixing warnings

* add migration tests that catch the bug

* make multiple migrations more robust

* add moved v9

* Merge branch 'noop-migrations' into single-db

* spurious line

* clean up migrations ordering

* update comment [ci skip]

* Bumping default number of max_open_files & re-ordering columns.

* fix merge

* fix ignored analysis tests

* Caching best block content

* Faster best_block_header

* Adding progress to v8 migration

* clean up warnings

* Separate hashes and bodies in the DB

* Separate hashes and bodies in the DB

* Fixed tests
2016-07-28 23:46:24 +02:00
Marek Kotewicz
f33cd60dc2 compaction profile used during migration, fixes #1750 (#1751)
* compaction profile used during migration, fixes #1750

* whitespace

[ci:skip]
2016-07-28 20:29:58 +02:00
keorn
02cf48681d Blocks and snapshot compression (#1687)
* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* new Compressible rlp trait

* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* DecompressingDecoder test

* initial compressing HashDB wrapper

* remove unused test

* change CompressedDB to struct wrapper with overlay

* simplify compressor

* failed RefCell attempt

* use denote to return reference

* compiled compresseddb

* compressdb test, add overlay emplace

* fix overlay reference count handling

* add immutable compresseddb, make account use hashdb

* simplify using trait objects

* enable hashdb for account

* initial state compression attempt

* wrap state db

* add tests for analyzing db

* add account predicate

* try to compress data fields as rlp too

* remove compression for storage trie

* add a compressing migration

* more compression stats tests

* fix migration import

* nested encoding compression test

* fix decompression, move db stats tests to rlpcompression

* added malformed rlp tests, cover a few edge cases

* new CompressingEncoder struct

* extend migrations to state

* first version working on the whole db

* clean up Compressible impl

* tests cleanup

* add a testing migration

* refactor deep compression using option, add simple compression

* put tests in a module

* fix compressed overlay loading

* simple compression for snapshots

* remove unused DecompressingDecoder

* add a general compressing migration

* add more common rlps to compress

* use static slices for swapper

* add precomputed hashes and invalid rlps

* make decoder private again

* cover more cases with tests

* style

* fix weird indentation

* remove possible panic in payload_info

* make prefix checking safe

* fix db existence check

* remove db dir from test

* pass usize by value [ci skip]

* Improve comment on panic removal.

* add common blocks db rlps

* add compression to blockchain db

* add blocks db migration

* fix the migrations

* remove state compression

* add a separate snapshot swapper

* ability to use different swappers and traversal

* update tests to new interface

* clean up code ordering

* update usage

* fix compilation

* remove unnecessary changes

* move methods to functions to reduce interface

* move test to module

* update common rlps to blocks db

* move tests to tests modules

* remove redundant &
2016-07-27 17:11:41 +02:00
Arkadiy Paronyan
9a8fdeead9 Enable WAL and disable DB repair (#1696)
* Enable WAL

* Disable rewind
2016-07-23 17:05:34 +02:00
Arkadiy Paronyan
4e447ccc68 More performance optimizations (#1649)
* Use tree index for DB

* Set uncles_hash, tx_root, receipts_root from verified block

* Use Filth instead of a bool

* Fix empty root check

* Flush block queue properly

* Expunge deref
2016-07-19 09:23:53 +02:00
Arkadiy Paronyan
5ab18d1313 Performance optimizations (#1642)
* Optimize ethash verification

* disable WAL for puts

* Clear account cache after commit

* Commit only modified accounts

* Optimize existing block check

* Cache last hashes
2016-07-17 09:18:15 +02:00
NikVolf
9fd95e6694 wiping also for non-prefixed db 2016-07-13 19:51:03 +02:00
NikVolf
9a0a5b4c22 disable wal & set block based cache when prefix specified 2016-07-13 19:05:06 +02:00
NikVolf
3ac33ceda0 block options for prefixed 2016-07-13 18:43:24 +02:00
Tomasz Drwięga
af65945b58 Specifying max open files (#1494) 2016-06-30 08:24:01 +02:00
Arkadiy Paronyan
a3a68a20f4 Merge pull request #1461 from ethcore/db-repair
Attempt DB repair if corrupted
2016-06-28 07:37:49 +02:00
NikVolf
f94f061f6a disable wal when commiting transactions 2016-06-27 22:12:23 +03:00
arkpar
ed153995f9 Attempt DB repair if corrupted 2016-06-27 20:41:02 +02:00
Nikolay Volf
9f4bfd9e7a fix tests 2016-06-27 18:47:50 +02:00
Nikolay Volf
2e5d5f12dd ethcore client config 2016-06-27 13:58:12 +02:00
Nikolay Volf
627b67db0a jdb to new settings config 2016-06-27 13:23:50 +02:00
Nikolay Volf
07098fd16f extra helpers for prefix 2016-06-27 13:14:40 +02:00
Nikolay Volf
e4763e90bc compaction struct and helpers 2016-06-27 13:03:34 +02:00
Gav Wood
4f1f33d1b8 Retweak BASE and MULTIPLIER in rocksdb config. (#1445) 2016-06-27 09:20:38 +02:00
Gav Wood
2400001675 More conservative settings for rocksdb. (#1440) 2016-06-25 23:13:34 +02:00
Nikolay Volf
27b18df3dd further rocksdb tuning (#1409) 2016-06-23 18:56:43 +02:00
NikVolf
9132895d0e limit flush rate 2016-06-21 17:20:47 +03:00
NikVolf
d42f8eac1e little less extreme file sizes 2016-06-21 12:07:07 +03:00
NikVolf
1f599ac6f9 fix type 2016-06-20 23:01:09 +03:00
NikVolf
d874555310 make default 100mb file size 2016-06-20 22:45:24 +03:00
Nikolay Volf
c6616431b0 fix tests 2016-06-20 14:20:13 +03:00
Nikolay Volf
7e452ab2e0 configurable cache size 2016-06-20 13:42:04 +03:00
Marek Kotewicz
ca008fb541 migration fixes (#1155)
* dont use system temp for migration, do backup

* fixed issue with number of open files on osx

* fixed compilation of util tests
2016-05-27 08:23:29 +02:00
Nikolay Volf
dfac17538f ethcore-db crate (#1097)
* trait

* implentated, lifetime issue still

* full api

* test mod

* working open

* get/retrieve

* fix warnings and bug

* working serialization of &[u8] parameters

* client attributes

* fix empty payload ser/de

* [ci skip] debug assert out

* extra deserialization test

* extra serialization test

* extra serialization test

* serialization fixes, nupdate rocksdb

* open test working

* result bug & remove some scaffolds

* fix warnings

* more simple tests

* consistent quotes

* get rid of dedicated is_open flag

* hashmap -> btreemap
2016-05-19 14:36:15 +02:00
Tomasz Drwięga
85da55a537 Fixing warnings 2016-04-06 10:55:40 +02:00
Tomasz Drwięga
8709dd28f8 Fixing clippy warnings 2016-03-11 10:58:51 +01:00
Tomasz Drwięga
cbc2c0cf0c Fixing clippy warnings. When building on nightly it is required to enable clippy 2016-03-07 14:34:17 +01:00
arkpar
764f6e9347 Merge branch 'db' of github.com:ethcore/parity into db 2016-02-22 13:48:43 +01:00
arkpar
077c5662a8 Fixed a warning 2016-02-22 13:47:25 +01:00
arkpar
67cd29f4e1 Whitespaces 2016-02-21 16:58:56 +01:00
Nikolay Volf
bcc4ca48ab to new namespace 2016-02-21 15:19:08 +03:00
arkpar
68795ea031 Fixed typos 2016-02-18 21:40:17 +01:00
arkpar
99600ba370 DB Test 2016-02-18 21:15:56 +01:00
arkpar
e99f604133 rocksdb abstraction layer 2016-02-18 03:46:34 +01:00