Commit Graph

789 Commits

Author SHA1 Message Date
Robert Habermeier
b18407b9e3 Snapshot optimizations (#1991)
* apply RLP compression to abridged blocks

* add memorydb consolidate

* code hash optimization

* add warning to snapshot restoration CLI
2016-08-25 14:28:45 +02:00
Marek Kotewicz
b0d462c6c9 Signature cleanup (#1921)
* Address renamed to H160 at bigint library level

* moved uint specific test from util to bigint library

* naming

* unifing hashes in progress

* unifing hashes

* cleanup redundant unwraps in tests

* Removing util/crypto in progress.

* fixed compiling

* signature cleanup in progress

* new module - ethcrypto used by ethstore and ethcore-network

* fixed compiling

* fixed compiling

* fixed merge
2016-08-24 18:35:21 +02:00
Robert Habermeier
190e4db266 Trie query recording and AccountDB factory for no mangling (#1944)
* optionally use no mangling for accountdb

* add the recorder module

* get_recorded for tries, no virtual dispatch on readonly tries

* add recording test
2016-08-24 16:53:36 +02:00
Tomasz Drwięga
33e0a234f2 Validating sha3 of a dapp bundle (#1993)
* Validating sha3 of a file

* sha3 in utils

* Removing devtools
2016-08-24 16:53:07 +02:00
Nikolay Volf
0c38197b43 Merge pull request #1956 from ethcore/db-cache
Apply settings to column families
2016-08-18 21:55:53 +04:00
Arkadiy Paronyan
8159d743ef ECIES without MAC (#1948)
* Use random IV for ECIES AES

* ECIES encrypt/decrypt for a single message

* Derive IV from shared secret
2016-08-18 18:08:02 +02:00
arkpar
f0cb835f44 Apply settings to column families 2016-08-18 18:04:54 +02:00
Robert Habermeier
57dbdaada9 remove impossible panickers related to infallible db transaction (#1947) 2016-08-18 09:43:56 +02:00
Tomasz Drwięga
bcf6b0b7d8 Bump clippy, fix warnings (#1939)
* Bumping clippy

* Fixing warnings
2016-08-17 16:06:41 +02:00
Arkadiy Paronyan
e2cf8a894f DB WAL size limit (#1935)
* Limit WAL size

* Check pruning by db modification date (#1924)
2016-08-17 15:55:26 +02:00
NikVolf
b45020fc7f add doc for address 2016-08-16 11:47:12 +03:00
Marek Kotewicz
c39761c042 ethkey and ethstore use hash structures from bigint (#1851)
* Address renamed to H160 at bigint library level

* moved uint specific test from util to bigint library

* naming

* unifing hashes in progress

* unifing hashes

* cleanup redundant unwraps in tests

* fixed compiling
2016-08-15 15:09:00 +02:00
Arkadiy Paronyan
0e7b06d3eb Save nodes removed from backing_overlay until commit (#1917) 2016-08-11 18:58:11 +02:00
Arkadiy Paronyan
417b70f90f Don't return deleted nodes that are not yet flushed (#1908) 2016-08-10 20:49:26 +02:00
Gav Wood
286b67d54b Wallet rpcs (#1898)
* Add wallet RPCs.

* Add wordlist file.

* Add standard brain wallet tests.

* Allow import of JSON wallets.

* Address grumble.
2016-08-10 17:57:40 +02:00
Tomasz Drwięga
a427208f79 Purging .derefs, fixing clippy warnings. (#1890)
* Fixing clippy warnings

* Purging derefs

* Simplifying engine derefs

* Simplifying more engine derefs
2016-08-10 16:29:40 +02:00
debris
974d537849 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-09 09:58:18 +02:00
Arkadiy Paronyan
7ed4bded52 Reduce max open files (#1876) 2016-08-08 17:18:29 +02:00
debris
8269887949 parse().unwrap() -> into() 2016-08-08 11:18:48 +02:00
debris
88c5f555a9 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-06 00:03:07 +02:00
Robert Habermeier
76a7246369 Snapshot creation and restoration (#1679)
* to_rlp takes self by-reference

* clean up some derefs

* out-of-order insertion for blockchain

* implement block rebuilder without verification

* group block chunk header into struct

* block rebuilder does verification

* integrate snapshot service with client service; flesh out implementation more

* initial implementation of snapshot service

* remove snapshottaker trait

* snapshot writer trait with packed and loose implementations

* write chunks using "snapshotwriter" in service

* have snapshot taking use snapshotwriter

* implement snapshot readers

* back up client dbs when replacing

* use snapshot reader in snapshot service

* describe offset format

* use new get_db_path in parity, allow some errors in service

* blockchain formatting

* implement parity snapshot

* implement snapshot restore

* force blocks to be submitted in order

* fix bug loading block hashes in packed reader

* fix seal field loading

* fix uncle hash computation

* fix a few bugs

* store genesis state in db. reverse block chunk order in packed writer

* allow out-of-order import for blocks

* bring restoration types together

* only snapshot the last 30000 blocks

* restore into overlaydb instead of journaldb

* commit version to database

* use memorydbs and commit directly

* fix trie test compilation

* fix failing tests

* sha3_null_rlp, not H256::zero

* move overlaydb to ref_overlaydb, add new overlaydb without on-disk rc

* port archivedb to new overlaydb

* add deletion mode tests for overlaydb

* use new overlaydb, check state root at end

* share chain info between state and block snapshotting

* create blocks snapshot using blockchain directly

* allow snapshot from arbitrary block, remove panickers from snapshot creation

* begin test framework

* blockchain chunking test

* implement stateproducer::tick

* state snapshot test

* create block and state chunks concurrently, better restoration informant

* fix tests

* add deletion mode tests for overlaydb

* address comments

* more tests

* Fix up tests.

* remove a few printlns

* add a little more documentation to `commit`

* fix tests

* fix ref_overlaydb test names

* snapshot command skeleton

* revert ref_overlaydb renaming

* reimplement snapshot commands

* fix many errors

* everything but inject

* get ethcore compiling

* get snapshot tests passing again

* instrument snapshot commands again

* fix fallout from other changes, mark snapshots as experimental

* optimize injection patterns

* do two injections

* fix up tests

* take snapshots from 1000 blocks efore

* address minor comments

* fix a few io crate related errors

* clarify names about total difficulty

[ci skip]
2016-08-05 17:00:46 +02:00
debris
316c9093db Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-05 10:53:34 +02:00
Arkadiy Paronyan
05bfdc508e Split IO and network crates (#1828)
* Abort on panic

* Split IO and network crates

* Restore panic handler

* Fixed doc tests
2016-08-05 10:32:04 +02:00
debris
df29fcff1a Merge branch 'split_hash' into move_hash 2016-08-05 09:56:36 +02:00
debris
1ed2de1d9b Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-05 09:30:01 +02:00
Robert Habermeier
b20833154e protect unsafety in plainhasher; do more hashing (#1841) 2016-08-05 09:15:55 +02:00
Robert Habermeier
8702a29e6f use mutex in dbtransaction (#1843) 2016-08-04 23:54:26 +02:00
debris
a1867a7ba6 bloomable trait 2016-08-04 15:38:16 +02:00
debris
24c7392500 removed redundant helper methods 2016-08-04 14:48:09 +02:00
debris
03c3d16744 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-08-04 08:41:30 +02:00
Arkadiy Paronyan
7093651d70 More performance optimizations (#1814)
* Buffered DB

* Use identity hash for MemoryDB

* Various tweaks

* Delayed DB compression

* Reduce last_hashes cloning

* Keep state cache

* Updating tests

* Optimized to_big_int

* Fixing build with stable

* Safer code
2016-08-03 22:03:40 +02:00
Tomasz Drwięga
deceb5fd56 Bumping clippy & fixing warnings (#1823)
* Bumping clippy

* Fixing clippy warnings

* Cargo.lock (latest nightly?)
2016-08-03 19:01:48 +02:00
Marek Kotewicz
531bc79edc removed unused code from util and unnecessary dependency of FixedHash (#1824) 2016-08-03 18:36:45 +02:00
Robert Habermeier
11b65ce53d Remove (almost all) panickers from trie module (#1776)
* memorydb ub patch and other cleanup

* fix denote invocations

* move trie traits into trie module

* replace "denote" with shim

* triedb returns results and no longer panics

* fix warnings

* get ethcore compiling

* warn on trie errors in ethcore

* remove unsafety from node decoder

* restore broken denote behavior for this branch

* fix overlayrecent fallout

* fix triedb tests

* remove unwrap in state

* alter Trie::get to return Result<Option<_>>

* fix refcell error in require

* fix test warnings

* fix json tests

* whitespace

[ci:skip]

* Avoid unneeded match/indentation

* whitespace

* prettify map_or_else

* remove test warning
2016-08-03 18:35:48 +02:00
debris
e8c451ac82 cleaning up hash reexports 2016-08-03 18:05:17 +02:00
debris
573e775ef9 Merge branch 'fixed_hash_util' into move_hash 2016-08-03 16:35:55 +02:00
Robert Habermeier
8c88e2a8cc JournalDB inject (#1806)
* add inject to journaldb

* adjust docs

* add test; fix refcounteddb impl

* fewer panics, fail on invalid insertions or deletions
2016-08-03 16:34:32 +02:00
debris
cad7125acc Merge branch 'master' of github.com:ethcore/parity into fixed_hash_util 2016-08-03 16:31:13 +02:00
debris
17bfc113c1 removed unused code from util and unnecessary dependency of FixedHash 2016-08-03 16:29:36 +02:00
Nikolay Volf
1b507e0147 Util & ipc clenup (#1807)
* removed frombytes stuff

* removed jsonrpc handler from nano
2016-08-02 15:02:47 +01:00
Marek Kotewicz
dbfc5cd31c fixed trace_transaction crash when block contained suicide (#1781) 2016-07-31 13:47:29 +02:00
debris
e9b3740ea8 Merge branch 'master' of github.com:ethcore/parity into move_hash 2016-07-30 14:14:25 +02:00
arkpar
8c47240248 Min and Max peers setting 2016-07-29 17:30:02 +02:00
Arkadiy Paronyan
57faa37623 Disable WAL (#1765)
* Disable WAL

* Make WAL optional

* Fix tests.

* Update cli.rs
2016-07-29 15:36:00 +02:00
Tomasz Drwięga
e4f0c0b215 Single DB (#1741)
* Consolidation migration

* Started db amalgamation

* Using client constants for columns

* Adding with_columns constructor

* Migrating to single db

* Fixing tests.

* test.sh without verbose

* Fixing warnings

* add migration tests that catch the bug

* make multiple migrations more robust

* add moved v9

* Merge branch 'noop-migrations' into single-db

* spurious line

* clean up migrations ordering

* update comment [ci skip]

* Bumping default number of max_open_files & re-ordering columns.

* fix merge

* fix ignored analysis tests

* Caching best block content

* Faster best_block_header

* Adding progress to v8 migration

* clean up warnings

* Separate hashes and bodies in the DB

* Separate hashes and bodies in the DB

* Fixed tests
2016-07-28 23:46:24 +02:00
Marek Kotewicz
f33cd60dc2 compaction profile used during migration, fixes #1750 (#1751)
* compaction profile used during migration, fixes #1750

* whitespace

[ci:skip]
2016-07-28 20:29:58 +02:00
keorn
30ba10e10c Better handling of multiple migrations (#1747)
* add migration tests that catch the bug

* make multiple migrations more robust

* clean up migrations ordering

* update comment [ci skip]

* remove explicit iter
2016-07-28 12:05:41 +02:00
keorn
02cf48681d Blocks and snapshot compression (#1687)
* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* new Compressible rlp trait

* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* DecompressingDecoder test

* initial compressing HashDB wrapper

* remove unused test

* change CompressedDB to struct wrapper with overlay

* simplify compressor

* failed RefCell attempt

* use denote to return reference

* compiled compresseddb

* compressdb test, add overlay emplace

* fix overlay reference count handling

* add immutable compresseddb, make account use hashdb

* simplify using trait objects

* enable hashdb for account

* initial state compression attempt

* wrap state db

* add tests for analyzing db

* add account predicate

* try to compress data fields as rlp too

* remove compression for storage trie

* add a compressing migration

* more compression stats tests

* fix migration import

* nested encoding compression test

* fix decompression, move db stats tests to rlpcompression

* added malformed rlp tests, cover a few edge cases

* new CompressingEncoder struct

* extend migrations to state

* first version working on the whole db

* clean up Compressible impl

* tests cleanup

* add a testing migration

* refactor deep compression using option, add simple compression

* put tests in a module

* fix compressed overlay loading

* simple compression for snapshots

* remove unused DecompressingDecoder

* add a general compressing migration

* add more common rlps to compress

* use static slices for swapper

* add precomputed hashes and invalid rlps

* make decoder private again

* cover more cases with tests

* style

* fix weird indentation

* remove possible panic in payload_info

* make prefix checking safe

* fix db existence check

* remove db dir from test

* pass usize by value [ci skip]

* Improve comment on panic removal.

* add common blocks db rlps

* add compression to blockchain db

* add blocks db migration

* fix the migrations

* remove state compression

* add a separate snapshot swapper

* ability to use different swappers and traversal

* update tests to new interface

* clean up code ordering

* update usage

* fix compilation

* remove unnecessary changes

* move methods to functions to reduce interface

* move test to module

* update common rlps to blocks db

* move tests to tests modules

* remove redundant &
2016-07-27 17:11:41 +02:00
Arkadiy Paronyan
c65ee93542 Use std::sync::Condvar (#1732)
test
2016-07-27 11:39:24 +02:00
Tomasz Drwięga
3f41186b2e Fixing some clippy warnings (#1728)
* Fixing warnings

* Fixing unnecessary ref

* Removing unnecessary operation
2016-07-26 20:31:25 +02:00