Commit Graph

151 Commits

Author SHA1 Message Date
Robert Habermeier
76a7246369 Snapshot creation and restoration (#1679)
* to_rlp takes self by-reference

* clean up some derefs

* out-of-order insertion for blockchain

* implement block rebuilder without verification

* group block chunk header into struct

* block rebuilder does verification

* integrate snapshot service with client service; flesh out implementation more

* initial implementation of snapshot service

* remove snapshottaker trait

* snapshot writer trait with packed and loose implementations

* write chunks using "snapshotwriter" in service

* have snapshot taking use snapshotwriter

* implement snapshot readers

* back up client dbs when replacing

* use snapshot reader in snapshot service

* describe offset format

* use new get_db_path in parity, allow some errors in service

* blockchain formatting

* implement parity snapshot

* implement snapshot restore

* force blocks to be submitted in order

* fix bug loading block hashes in packed reader

* fix seal field loading

* fix uncle hash computation

* fix a few bugs

* store genesis state in db. reverse block chunk order in packed writer

* allow out-of-order import for blocks

* bring restoration types together

* only snapshot the last 30000 blocks

* restore into overlaydb instead of journaldb

* commit version to database

* use memorydbs and commit directly

* fix trie test compilation

* fix failing tests

* sha3_null_rlp, not H256::zero

* move overlaydb to ref_overlaydb, add new overlaydb without on-disk rc

* port archivedb to new overlaydb

* add deletion mode tests for overlaydb

* use new overlaydb, check state root at end

* share chain info between state and block snapshotting

* create blocks snapshot using blockchain directly

* allow snapshot from arbitrary block, remove panickers from snapshot creation

* begin test framework

* blockchain chunking test

* implement stateproducer::tick

* state snapshot test

* create block and state chunks concurrently, better restoration informant

* fix tests

* add deletion mode tests for overlaydb

* address comments

* more tests

* Fix up tests.

* remove a few printlns

* add a little more documentation to `commit`

* fix tests

* fix ref_overlaydb test names

* snapshot command skeleton

* revert ref_overlaydb renaming

* reimplement snapshot commands

* fix many errors

* everything but inject

* get ethcore compiling

* get snapshot tests passing again

* instrument snapshot commands again

* fix fallout from other changes, mark snapshots as experimental

* optimize injection patterns

* do two injections

* fix up tests

* take snapshots from 1000 blocks efore

* address minor comments

* fix a few io crate related errors

* clarify names about total difficulty

[ci skip]
2016-08-05 17:00:46 +02:00
debris
03c3d16744 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-04 08:41:30 +02:00
Arkadiy Paronyan
7093651d70 More performance optimizations (#1814)
* Buffered DB

* Use identity hash for MemoryDB

* Various tweaks

* Delayed DB compression

* Reduce last_hashes cloning

* Keep state cache

* Updating tests

* Optimized to_big_int

* Fixing build with stable

* Safer code
2016-08-03 22:03:40 +02:00
Tomasz Drwięga
deceb5fd56 Bumping clippy & fixing warnings (#1823)
* Bumping clippy

* Fixing clippy warnings

* Cargo.lock (latest nightly?)
2016-08-03 19:01:48 +02:00
debris
e8c451ac82 cleaning up hash reexports 2016-08-03 18:05:17 +02:00
Arkadiy Paronyan
087ebcf94e Fixing cache update after reorg (#1816) 2016-08-03 10:35:04 +02:00
Arkadiy Paronyan
d891e80ad7 Commit best block after closing transaction (#1791) 2016-08-01 10:10:13 -07:00
Marek Kotewicz
bcf8cd6dc0 cache manager and clearing tracing cache (#1769)
* removed configure_cache method

* generic cache_manager struct

* fixed #1743, tracing caches are cleared

* removed deadlocks in garbage_collect, implemented HeapSizeOf for traces

* trace cache config

* fixed carbage typo
2016-07-30 15:19:27 -07:00
Gav Wood
4304f4a2a4 Minor tweak to logging. Fix CLI help. 2016-07-29 00:22:46 +02:00
Tomasz Drwięga
e4f0c0b215 Single DB (#1741)
* Consolidation migration

* Started db amalgamation

* Using client constants for columns

* Adding with_columns constructor

* Migrating to single db

* Fixing tests.

* test.sh without verbose

* Fixing warnings

* add migration tests that catch the bug

* make multiple migrations more robust

* add moved v9

* Merge branch 'noop-migrations' into single-db

* spurious line

* clean up migrations ordering

* update comment [ci skip]

* Bumping default number of max_open_files & re-ordering columns.

* fix merge

* fix ignored analysis tests

* Caching best block content

* Faster best_block_header

* Adding progress to v8 migration

* clean up warnings

* Separate hashes and bodies in the DB

* Separate hashes and bodies in the DB

* Fixed tests
2016-07-28 23:46:24 +02:00
Gav Wood
0934a283b2 Tweak format of log so it's not so verbose. (#1758) 2016-07-28 23:45:56 +02:00
keorn
02cf48681d Blocks and snapshot compression (#1687)
* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* new Compressible rlp trait

* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* DecompressingDecoder test

* initial compressing HashDB wrapper

* remove unused test

* change CompressedDB to struct wrapper with overlay

* simplify compressor

* failed RefCell attempt

* use denote to return reference

* compiled compresseddb

* compressdb test, add overlay emplace

* fix overlay reference count handling

* add immutable compresseddb, make account use hashdb

* simplify using trait objects

* enable hashdb for account

* initial state compression attempt

* wrap state db

* add tests for analyzing db

* add account predicate

* try to compress data fields as rlp too

* remove compression for storage trie

* add a compressing migration

* more compression stats tests

* fix migration import

* nested encoding compression test

* fix decompression, move db stats tests to rlpcompression

* added malformed rlp tests, cover a few edge cases

* new CompressingEncoder struct

* extend migrations to state

* first version working on the whole db

* clean up Compressible impl

* tests cleanup

* add a testing migration

* refactor deep compression using option, add simple compression

* put tests in a module

* fix compressed overlay loading

* simple compression for snapshots

* remove unused DecompressingDecoder

* add a general compressing migration

* add more common rlps to compress

* use static slices for swapper

* add precomputed hashes and invalid rlps

* make decoder private again

* cover more cases with tests

* style

* fix weird indentation

* remove possible panic in payload_info

* make prefix checking safe

* fix db existence check

* remove db dir from test

* pass usize by value [ci skip]

* Improve comment on panic removal.

* add common blocks db rlps

* add compression to blockchain db

* add blocks db migration

* fix the migrations

* remove state compression

* add a separate snapshot swapper

* ability to use different swappers and traversal

* update tests to new interface

* clean up code ordering

* update usage

* fix compilation

* remove unnecessary changes

* move methods to functions to reduce interface

* move test to module

* update common rlps to blocks db

* move tests to tests modules

* remove redundant &
2016-07-27 17:11:41 +02:00
Tomasz Drwięga
3f41186b2e Fixing some clippy warnings (#1728)
* Fixing warnings

* Fixing unnecessary ref

* Removing unnecessary operation
2016-07-26 20:31:25 +02:00
Arkadiy Paronyan
22a87b1203 Use transaction for writing blocks (#1718) 2016-07-26 00:25:49 +02:00
Gav Wood
856657e39a Log a chain-reorg. (#1715)
* Log a chain-reorg.

* Nicer output

* Use imported rather than enacted.

Enacted can include previously imported blocks which makes the
info incorrect.
2016-07-26 00:20:37 +02:00
Arkadiy Paronyan
898f1410a5 Rotate blockchain cache (#1709) 2016-07-25 18:17:45 +02:00
Marek Kotewicz
226fe8e0bb cli overhaul (#1600)
* cli commands

* cleanup parity/signer

* cleanup parity/signer

* remove redundant import of signer crate from main.rs

* cli cleanup in progress

* cli cleanup in progress

* moved few commonly used functions to separate methods with tests

* cleaning up blockchain import in progress

* cleaning up blockchain import in progress2

* cleaning up blockchain import in progress3

* tests for database compaction profile parsing

* cleaning up blockchain import in progress4

* cleaning up blockchain import in progress5

* blockchain import

* export blockchain in progress

* cleanup execute_export

* Configuration::to_duration cleaned up

* removed unused code, tests for to_duration

* cleanup Configuration::mode function

* parsing some of the cli params in params.rs

* rpc and signer are no longer optional

* move importing extern crates to main.rs file

* swipe dies from rpc module

* swipe dies from dapps

* finding deprecated

* several tests and fixes for parity

* parity cleanup in progress

* cleanup price parsing

* parity cleanup in progress

* swiped all dies

* parity cleanup in progress

* replace usages of from_str with parse() in parity/params.rs

* removed few more from_str

* split parity/params.rs into params and helpers

* removed wildcard import from configuration.rs

* cleanup directories/path creation

* cleaning up run cmd

* moved LoggerConfig

* defaults for cli params

* fixed indention in raise_fd_limit

* tests for rpc_apis

* tests for default ipc and rpc settings

* ipc socket

* cleanup in progress

* account service

* cleanup miner config

* BlockChain commands use Directiores structure now

* client_config

* network settings and dapps configuration

* removing warnings

* default logger config

* fixed client_path

* overhaul

* fixing export && import

* default export DataFormat

* import and export also upgrade db

* fixed export && import

* polishing pr

* polishing pr

* fixed custom bootnodes

* fixed daemonize on windows

* fixed setting up enable network

* finished pr

* fixed compiling on windows

* Fixed warning; windows build

* Better cache management

* Fixed tests on windows

* Fixed test

* Restored pruning method names

* --cache alias

* Fixed more tests

* Ensure default options actually listed as valid

[ci:skip]
2016-07-25 16:09:47 +02:00
Arkadiy Paronyan
f048839a4b Fixing warnings (#1705) 2016-07-25 10:21:02 +02:00
Arkadiy Paronyan
9a8fdeead9 Enable WAL and disable DB repair (#1696)
* Enable WAL

* Disable rewind
2016-07-23 17:05:34 +02:00
gregg dourgarian
d67369a01c fix typos (#1644) 2016-07-19 20:42:23 +02:00
Tomasz Drwięga
038862fa9d Fixing clippy warnings (#1660) 2016-07-19 09:25:51 +02:00
Arkadiy Paronyan
4e447ccc68 More performance optimizations (#1649)
* Use tree index for DB

* Set uncles_hash, tx_root, receipts_root from verified block

* Use Filth instead of a bool

* Fix empty root check

* Flush block queue properly

* Expunge deref
2016-07-19 09:23:53 +02:00
Arkadiy Paronyan
dd17c766b8 Blockchain repair on missing state root (#1646)
* BC locking; Recovery batch

* Missing state root recovery

* Test
2016-07-17 23:03:29 +02:00
Arkadiy Paronyan
5ab18d1313 Performance optimizations (#1642)
* Optimize ethash verification

* disable WAL for puts

* Clear account cache after commit

* Commit only modified accounts

* Optimize existing block check

* Cache last hashes
2016-07-17 09:18:15 +02:00
Robert Habermeier
d5d0cb0b47 don't batch best block for branches (#1623) 2016-07-14 19:16:01 +02:00
Nikolay Volf
50c43bd819 Merge pull request #1610 from ethcore/db-repair
Fix bloomchain on blockchain repair
2016-07-14 01:14:57 +02:00
Robert Habermeier
36d3d0d7d7 replace synchronization primitives with those from parking_lot (#1593)
* parking_lot in cargo.toml

* replace all lock invocations with parking_lot ones

* use parking_lot synchronization primitives
2016-07-13 19:59:59 +02:00
arkpar
80d59949ff Fix bloomchain on blockchain repair 2016-07-13 14:28:46 +02:00
Gav Wood
9f43526c88 Merge branch 'master' into kill_unwraps 2016-07-07 09:40:12 +02:00
Nikolay Volf
8282c7dd50 Client IPC Interface (#1493)
* btree map serializer

* serde tests

* state diff serialization

* basic layout

* more missing serializaers

* uncle returns rlp

* block queue info

* sorting with transaction result

* sorting out util imports

* transaction import result sorting also

* sorting filters & ranges

* error sorting out

* deriving ipc service compiling

* rpc & sync recompile

* sorting rpc using uncles

* fix compilation

* fix merging bugs

* fix unused imports

* fix all warnings

* tests stub

* some merge bugs

* ethcore compilation

* fix rpc compilation

* deriving attribute

* tests (and fixes)

* rpc test working

* fix warnings again

* rs.in -> rs

* missing attribute

* refactored tree changes

* paste reformat mess fix

* pub mod actually

* intendation fix
2016-07-07 09:39:32 +02:00
Gav Wood
3b662c285f Switch out .X().unwrap() for .unwrapped_X 2016-07-07 09:37:31 +02:00
arkpar
548b4e164e Attempt to fix blochchain DB sync 2016-07-04 18:24:14 +02:00
Nikolay Volf
7e452ab2e0 configurable cache size 2016-06-20 13:42:04 +03:00
Marek Kotewicz
7370776af1 Bloomchain (#1014)
* use bloomchain crate in blockchain module. remove obsole chainfilter submodule

* update database version to 6.0

* removed redundant line

* simple db migration

* make migration slightly more functional

* bloomchain migration

* migration version is just a single unsigned integer

* updated migration v6

* parity migration

* db migration

* removed hardcoded migration dir

* replace ptr::copy with clone_from_slice, removed potential endianess problem from trace/db.rs

* removed superfluous line

* blockchains log blooms config is not exposed any more
2016-05-26 18:24:51 +02:00
Arkadiy Paronyan
fba5082b00 Propagate uncles (#1134) 2016-05-24 21:56:17 +02:00
Nikolay Volf
4e41cbca81 Moving all Client public API types to separate mod & binary serialization codegen for that mod (#1051)
* transaction moved

* trash remove

* ids move

* receipt

* tree-route

* blockchain info

* log_entry move

* trace filter moved

* executed & trace moved

* localized trace moved

* block status moved

* build scripts and codegen refs

* Cargo.lock update

* binary for blockstatus, blockchaininfo

* binary for trace

* trace filters binary ser

* binary for log entries & executed

* binary for receipt

* special case for u8 & transaction binary attribute

* resolved remaining issues & error binary serialization

* json-tests util import

* fix warnings

* ids attr

* add missing attributes

* Update build.rs
2016-05-16 18:33:32 +02:00
Marek Kotewicz
66477a9476 Core tracedb functionality. (#996)
* fixed encoding 0u8

* simplified if else stmt

* tracedb core

* more comprehensive tracedb tests

* fixed minor review issues

* addresses filter

* fixed typos

* replace malformed with corrupted

* trace switch

* db key is generic and can be made smaller

* smaller tracedb keys

* tracedb version

* fixed ignored tests

* rename Tracedb -> TraceDB

* fixed typos

* proves

* trace only top level calls to builtins to avoid DDoS attacks

* fixed tracedb config switches

* fix comments fat replaced with trace

* vector-addressing scheme for localized traces

* removed comments

* removed first, redundant 0 from trace address

* updated db.trace method

* additional tests for tracedb.trace()
2016-04-30 16:41:24 +01:00
Gav Wood
242b986196 Fix spelling of "omitted" 2016-04-24 22:16:34 +01:00
Gav Wood
0810de4415 Fix spelling of "omitted" 2016-04-24 22:16:06 +01:00
debris
d5555de1b9 import route contains ommited blocks 2016-04-24 19:40:04 +02:00
Arkadiy Paronyan
f2a5630fdf Merge pull request #972 from ethcore/db_writer
querying extras separated to its own module
2016-04-21 11:26:13 +02:00
Arkadiy Paronyan
e47af7f745 Merge pull request #966 from ethcore/from-bytes-extend
Addressing binary serialization for db types
2016-04-20 18:17:00 +02:00
debris
bffa1e1ec9 simplified writing and reading from database with cache 2016-04-20 15:45:42 +02:00
debris
62455a4094 separated from blockchain and made reusable db reader and batch writer 2016-04-18 18:15:03 +02:00
debris
524a495ffd temporary fix of panic in blockchain garbage collection 2016-04-18 15:18:14 +02:00
Nikolay Volf
17f26ad588 finished for BlockLocation 2016-04-17 18:30:42 +03:00
Nikolay Volf
bd2149406d from/to for BlockLocation 2016-04-17 18:18:25 +03:00
debris
295efdba55 refactored blockchain extras keys building 2016-04-15 18:54:35 +02:00
Tomasz Drwięga
85da55a537 Fixing warnings 2016-04-06 10:55:40 +02:00
debris
2a3e695f8a LocalizedReceipt 2016-03-20 17:29:39 +01:00
Gav Wood
66837452c2 Expose tracing all the way to BlockChain; now it's up to blockchain to integrate. 2016-03-19 13:37:47 +01:00
Gav Wood
59d0d2df9a Don't change best block until extras is committed. 2016-03-15 10:59:58 +01:00
Gav Wood
b25551be7b Merge pull request #671 from ethcore/clippy_hook
Removing running clippy by default on nightly.
2016-03-11 14:03:16 +01:00
Tomasz Drwięga
d84e008e00 Removing superflous check for nightly 2016-03-11 11:16:49 +01:00
debris
bfac5f9238 Merge branch 'master' of github.com:ethcore/parity into client_submodules 2016-03-11 11:03:52 +01:00
debris
a6d268db16 fixed missing reexport 2016-03-11 10:30:13 +01:00
arkpar
5f37f6edb4 Correct cache update order 2016-03-10 21:01:17 +01:00
arkpar
a6b5aad8c9 Merge branch 'master' of github.com:ethcore/parity into thread 2016-03-10 18:40:17 +01:00
Gav Wood
16e12b1ae5 Merge pull request #645 from ethcore/import_route
blockchain import_route
2016-03-10 13:14:56 +01:00
debris
0a7cda09ff changed route name to enacted and retracted 2016-03-10 10:17:17 +01:00
debris
f397fb210f fixed typo 2016-03-10 00:11:35 +01:00
debris
8b042ac875 blockchain import_route 2016-03-09 21:55:23 +01:00
arkpar
57485a73ec Merge branch 'master' into thread 2016-03-07 15:10:15 +01:00
Tomasz Drwięga
ab42ec8c81 Removing unneeded lifetime 2016-03-07 14:40:39 +01:00
Tomasz Drwięga
cbc2c0cf0c Fixing clippy warnings. When building on nightly it is required to enable clippy 2016-03-07 14:34:17 +01:00
Gav Wood
52dceff3ea Merge branch 'master' into finduncles 2016-03-03 15:16:56 +01:00
Gav Wood
f8dc1f2e3a Avoid leaking block_details.
Fixes #576
2016-03-03 12:56:34 +01:00
Gav Wood
dadc2a96ea shrink_to_fit after removing hashes. 2016-03-03 12:39:19 +01:00
Gav Wood
6933bb971b Test. 2016-03-02 23:41:15 +01:00
Gav Wood
039c0056bc Uncle inclusion in block authoring. Still need tests. 2016-03-02 19:38:00 +01:00
Gav Wood
877270c35f Fixes. 2016-03-02 18:32:54 +01:00
Gav Wood
42df98450c Include uncles in exclused. 2016-03-02 18:05:47 +01:00
Gav Wood
671965d44f Test for ancestry. 2016-03-02 17:31:42 +01:00
Gav Wood
c75737bcf0 Add ancestry iterator. 2016-03-02 17:04:44 +01:00
Gav Wood
162300a4a6 Merge pull request #547 from ethcore/mining
Mining
2016-03-02 13:00:22 +01:00
Nikolay Volf
6b9fde5790 Merge branch 'master' into mining 2016-03-02 08:54:21 +03:00
debris
67a1f2065e fixed broken master 2016-03-02 06:01:31 +01:00
debris
32074cc420 fixed compilation issue caused by incorrect merge 2016-03-02 05:55:42 +01:00
debris
4571df9fc6 Merge branch 'master' of github.com:ethcore/parity into chain_generator 2016-03-02 04:54:55 +01:00
debris
010659cf12 added missing , in comment 2016-03-02 04:54:47 +01:00
debris
68ba016279 improved blockchain generator 2016-03-02 04:25:03 +01:00
arkpar
15ecba57ec Merge branch 'master' of github.com:ethcore/parity into thread 2016-03-02 01:20:41 +01:00
Gav Wood
d55aa54412 Merge branch 'master' into mining 2016-03-02 00:59:39 +01:00
Gav Wood
0082d12ea9 Merge pull request #544 from ethcore/uint-refact
uint to separate crate
2016-03-02 00:59:03 +01:00
Gav Wood
140711dd8a Fixups from review.
Reduce size of default extra-data.
Introduce find_uncle_headers.
2016-03-01 19:59:12 +01:00
debris
3e2366b38b improved chain generator 2016-03-01 16:22:06 +01:00
debris
47688e49cd removed redundant whitespaces 2016-03-01 13:54:14 +01:00
debris
063020f507 added license headers 2016-03-01 13:46:33 +01:00
debris
ab9fddf6b2 blockchain generator 2016-03-01 13:44:09 +01:00
Nikolay Volf
f528d8c50a common -> numbers (as most narrow) 2016-03-01 00:44:45 +03:00
Nikolay Volf
d0125f3ff5 uint to separated crate 2016-03-01 00:21:15 +03:00
arkpar
d0129ff67b Fixed cache memory leak 2016-02-29 21:15:39 +01:00
arkpar
cb4d17825b Fixed lock order 2016-02-29 19:49:29 +01:00
arkpar
0ccbba9073 Merge branch 'master' of github.com:ethcore/parity into thread 2016-02-29 18:11:59 +01:00
debris
1481f3f477 replaced match with if to shorten the code 2016-02-27 19:27:34 +01:00
debris
1cc719d413 description for prepare_block_blooms_update function 2016-02-27 19:17:29 +01:00
debris
53b19e9f2d Merge branch 'master' of https://github.com/ethcore/parity into blockchain_cleanup 2016-02-27 16:06:02 +01:00
debris
7b3613e1f0 updated ExtrasUpdate function 2016-02-27 10:21:44 +01:00
debris
01f69ca80c moved creation of blockchains db transaction to apply_update function 2016-02-27 10:19:33 +01:00
debris
cd43e32e25 added docs and license headers 2016-02-27 02:16:39 +01:00