Commit Graph

102 Commits

Author SHA1 Message Date
Nikolay Volf
d315ec29e1 Apply pending block details on commit (#2254)
* failing test

* Cache pending details

* [ci skip] updated comment
2016-10-27 15:26:29 +02:00
Tomasz Drwięga
88997801d0 Clippy bump (#2877)
* Bumping clippy

* Fixing warnings

* Fix the "fix"
2016-10-27 08:28:12 +02:00
Robert Habermeier
bc81ae0407 Snapshot and blockchain stability improvements (#2843)
* allow taking snapshot from just-restored database without error

* make creation informant less spammy

* Ancestry iterator failure-resilient

* make uncle hash searching resilient to incomplete chain

* deduce pre-chunk info from last written block's details
2016-10-24 18:27:23 +02:00
Robert Habermeier
96f4c10453 sweep most unwraps from ethcore crate, dapps crate (#2762)
* sweep most unwraps from ethcore crate

* purge unwrap from dapps server

* whitespace

[ci:none]
2016-10-20 23:41:15 +02:00
Arkadiy Paronyan
487dfb0208 Snapshot sync part 2 (#2098)
* Split block downloader into a module

* Snapshot sync progress

* Warp sync CLI option

* Increased snapshot chunk and ping timeouts

* Fixed an issue with delayed writes

* Updated bootnodes

* Don't run pending IO tasks on shutdown

* Optional first_block; removed insert_snapshot_block

* Fixing expect calls

* Fixed stalled sync

* style and docs

* Update block_sync.rs

[ci:skip]
2016-10-18 18:16:00 +02:00
Gav Wood
85eeb3ea6e EIP150.1c (#2591)
* EIP150.1b first part.

* Add documentation.

[ci:skip]

* Tweak comment

* Fix some tests.

* Fix create

* Add the new EIP150 test. (#2563)

* Fix create some more.

* Another fix for create.

* Final fixes.

* Enable EIP150 test.

* Revert the costly suicides to respect EIP150.1b.

* Fixups for EIP150.1c

* Update tests.

* Updating tests

* Adding state tests

* Updating tests

* Updating HF block

* Removing flushln [ci:skip]

* Commenting out suicide test

* New tests.

* Switch to EIP tests.

* Rename some stuff, combine transition tests.

* Fix U256 overflows.

* Name cleanups and an effort to fix tests.

* Rename back for clarity.

* Fix non-EIP-150 OOG CALLs

* New transition block number & additional EIP150 tests

* Morden transition block set; Tests updated

* Added missing test
2016-10-15 14:39:15 +02:00
Tomasz Drwięga
443eb19739 Fixing logs-receipt matching (#2403) 2016-09-30 11:14:30 +02:00
Tomasz Drwięga
01018b417a Fixing transaction queue (#2392) 2016-09-29 12:46:04 +02:00
Tomasz Drwięga
baa2feaca6 Removing extras data from retracted blocks. (#2375)
* Removing extras data from retracted blocks.

* Adding a test case
2016-09-28 15:49:42 +02:00
Tomasz Drwięga
7f7e74869c Bump clippy & Fix warnings (#2109)
* Bump clippy

* Fixing warnings
2016-09-16 23:03:26 +02:00
Tomasz Drwięga
9ed9857fba Logs limit & log_index bug (#2073)
* Limiting number of logs

* Test for logs

* Fixing logs ordering and indexing

* Fixing sort

* unwrap -> expect

* Revert "unwrap -> expect"

This reverts commit e99e6e77f37692fe568448e768aa72775de8d0cd.
2016-09-14 12:02:30 +02:00
Robert Habermeier
5dd56aa070 port ethcore 2016-09-01 14:29:59 +02:00
debris
f5f4736e7c simplified iterator 2016-08-31 17:36:49 +02:00
debris
996b4b9dc0 fixed transaction addresses mapping, fixes #1971 2016-08-31 16:55:43 +02:00
Nipunn Koorapati
4389742ca3 Make the block header struct's internals private (#2000)
* Make the block header struct's internals private

Currently, this involves a lot of explicit cloning, but we
could migrate the return types of the get_* functions to
be copies rather than references since they are mostly copy
types anyway.

I opted to eliminate the constructor in favor of using
Default::default() plus calling a bunch of setters. This
is similar to the model that a Google Protobuf client uses
and I think it looks fine.

* Drop some unnecessary cloning by comparing references

* Fix compiler errors from callsites in tests.
2016-08-29 11:35:23 +02:00
Robert Habermeier
2aef81cf90 remove internal locking from DBTransaction (#2003) 2016-08-25 16:43:56 +02:00
debris
b96d4eaddb bloom optimization 2016-08-24 13:59:50 +02:00
Robert Habermeier
0e0cc20d84 keep track of first block in blockchain (#1937) 2016-08-18 22:01:57 +02:00
Robert Habermeier
9adf77581c move column family constants into db module (#1955) 2016-08-18 18:24:49 +02:00
Robert Habermeier
57dbdaada9 remove impossible panickers related to infallible db transaction (#1947) 2016-08-18 09:43:56 +02:00
Robert Habermeier
9732da5101 replace cache manager with mutex; reduce lock hold times 2016-08-08 16:25:48 +02:00
debris
c5a0024eeb fixed cache_manager lock order 2016-08-08 16:14:37 +02:00
Arkadiy Paronyan
35451e31d4 Update cache usage on commiting block info (#1871) 2016-08-08 13:47:00 +02:00
Robert Habermeier
76a7246369 Snapshot creation and restoration (#1679)
* to_rlp takes self by-reference

* clean up some derefs

* out-of-order insertion for blockchain

* implement block rebuilder without verification

* group block chunk header into struct

* block rebuilder does verification

* integrate snapshot service with client service; flesh out implementation more

* initial implementation of snapshot service

* remove snapshottaker trait

* snapshot writer trait with packed and loose implementations

* write chunks using "snapshotwriter" in service

* have snapshot taking use snapshotwriter

* implement snapshot readers

* back up client dbs when replacing

* use snapshot reader in snapshot service

* describe offset format

* use new get_db_path in parity, allow some errors in service

* blockchain formatting

* implement parity snapshot

* implement snapshot restore

* force blocks to be submitted in order

* fix bug loading block hashes in packed reader

* fix seal field loading

* fix uncle hash computation

* fix a few bugs

* store genesis state in db. reverse block chunk order in packed writer

* allow out-of-order import for blocks

* bring restoration types together

* only snapshot the last 30000 blocks

* restore into overlaydb instead of journaldb

* commit version to database

* use memorydbs and commit directly

* fix trie test compilation

* fix failing tests

* sha3_null_rlp, not H256::zero

* move overlaydb to ref_overlaydb, add new overlaydb without on-disk rc

* port archivedb to new overlaydb

* add deletion mode tests for overlaydb

* use new overlaydb, check state root at end

* share chain info between state and block snapshotting

* create blocks snapshot using blockchain directly

* allow snapshot from arbitrary block, remove panickers from snapshot creation

* begin test framework

* blockchain chunking test

* implement stateproducer::tick

* state snapshot test

* create block and state chunks concurrently, better restoration informant

* fix tests

* add deletion mode tests for overlaydb

* address comments

* more tests

* Fix up tests.

* remove a few printlns

* add a little more documentation to `commit`

* fix tests

* fix ref_overlaydb test names

* snapshot command skeleton

* revert ref_overlaydb renaming

* reimplement snapshot commands

* fix many errors

* everything but inject

* get ethcore compiling

* get snapshot tests passing again

* instrument snapshot commands again

* fix fallout from other changes, mark snapshots as experimental

* optimize injection patterns

* do two injections

* fix up tests

* take snapshots from 1000 blocks efore

* address minor comments

* fix a few io crate related errors

* clarify names about total difficulty

[ci skip]
2016-08-05 17:00:46 +02:00
Arkadiy Paronyan
7093651d70 More performance optimizations (#1814)
* Buffered DB

* Use identity hash for MemoryDB

* Various tweaks

* Delayed DB compression

* Reduce last_hashes cloning

* Keep state cache

* Updating tests

* Optimized to_big_int

* Fixing build with stable

* Safer code
2016-08-03 22:03:40 +02:00
Tomasz Drwięga
deceb5fd56 Bumping clippy & fixing warnings (#1823)
* Bumping clippy

* Fixing clippy warnings

* Cargo.lock (latest nightly?)
2016-08-03 19:01:48 +02:00
Arkadiy Paronyan
087ebcf94e Fixing cache update after reorg (#1816) 2016-08-03 10:35:04 +02:00
Arkadiy Paronyan
d891e80ad7 Commit best block after closing transaction (#1791) 2016-08-01 10:10:13 -07:00
Marek Kotewicz
bcf8cd6dc0 cache manager and clearing tracing cache (#1769)
* removed configure_cache method

* generic cache_manager struct

* fixed #1743, tracing caches are cleared

* removed deadlocks in garbage_collect, implemented HeapSizeOf for traces

* trace cache config

* fixed carbage typo
2016-07-30 15:19:27 -07:00
Gav Wood
4304f4a2a4 Minor tweak to logging. Fix CLI help. 2016-07-29 00:22:46 +02:00
Tomasz Drwięga
e4f0c0b215 Single DB (#1741)
* Consolidation migration

* Started db amalgamation

* Using client constants for columns

* Adding with_columns constructor

* Migrating to single db

* Fixing tests.

* test.sh without verbose

* Fixing warnings

* add migration tests that catch the bug

* make multiple migrations more robust

* add moved v9

* Merge branch 'noop-migrations' into single-db

* spurious line

* clean up migrations ordering

* update comment [ci skip]

* Bumping default number of max_open_files & re-ordering columns.

* fix merge

* fix ignored analysis tests

* Caching best block content

* Faster best_block_header

* Adding progress to v8 migration

* clean up warnings

* Separate hashes and bodies in the DB

* Separate hashes and bodies in the DB

* Fixed tests
2016-07-28 23:46:24 +02:00
Gav Wood
0934a283b2 Tweak format of log so it's not so verbose. (#1758) 2016-07-28 23:45:56 +02:00
keorn
02cf48681d Blocks and snapshot compression (#1687)
* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* new Compressible rlp trait

* new Compressible rlp trait

* new Compressible rlp trait

* make compressed rlp iterable

* make compressed rlp iterable

* make compressed rlp iterable

* invalid rlp slice swapper

* invalid rlp slice swapper

* invalid rlp slice swapper

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* switch compress to swapper, add reverse swapper test case

* add basic account compression test

* add new rlp trait

* add account compress/ decompress test

* make compressor cleaner, use hashmaps for swapper

* improve compression tests

* add a DecompressingDecoder, change Decoder to take refernce

* separate rlp compression related stuff

* DecompressingDecoder test

* initial compressing HashDB wrapper

* remove unused test

* change CompressedDB to struct wrapper with overlay

* simplify compressor

* failed RefCell attempt

* use denote to return reference

* compiled compresseddb

* compressdb test, add overlay emplace

* fix overlay reference count handling

* add immutable compresseddb, make account use hashdb

* simplify using trait objects

* enable hashdb for account

* initial state compression attempt

* wrap state db

* add tests for analyzing db

* add account predicate

* try to compress data fields as rlp too

* remove compression for storage trie

* add a compressing migration

* more compression stats tests

* fix migration import

* nested encoding compression test

* fix decompression, move db stats tests to rlpcompression

* added malformed rlp tests, cover a few edge cases

* new CompressingEncoder struct

* extend migrations to state

* first version working on the whole db

* clean up Compressible impl

* tests cleanup

* add a testing migration

* refactor deep compression using option, add simple compression

* put tests in a module

* fix compressed overlay loading

* simple compression for snapshots

* remove unused DecompressingDecoder

* add a general compressing migration

* add more common rlps to compress

* use static slices for swapper

* add precomputed hashes and invalid rlps

* make decoder private again

* cover more cases with tests

* style

* fix weird indentation

* remove possible panic in payload_info

* make prefix checking safe

* fix db existence check

* remove db dir from test

* pass usize by value [ci skip]

* Improve comment on panic removal.

* add common blocks db rlps

* add compression to blockchain db

* add blocks db migration

* fix the migrations

* remove state compression

* add a separate snapshot swapper

* ability to use different swappers and traversal

* update tests to new interface

* clean up code ordering

* update usage

* fix compilation

* remove unnecessary changes

* move methods to functions to reduce interface

* move test to module

* update common rlps to blocks db

* move tests to tests modules

* remove redundant &
2016-07-27 17:11:41 +02:00
Tomasz Drwięga
3f41186b2e Fixing some clippy warnings (#1728)
* Fixing warnings

* Fixing unnecessary ref

* Removing unnecessary operation
2016-07-26 20:31:25 +02:00
Arkadiy Paronyan
22a87b1203 Use transaction for writing blocks (#1718) 2016-07-26 00:25:49 +02:00
Gav Wood
856657e39a Log a chain-reorg. (#1715)
* Log a chain-reorg.

* Nicer output

* Use imported rather than enacted.

Enacted can include previously imported blocks which makes the
info incorrect.
2016-07-26 00:20:37 +02:00
Arkadiy Paronyan
898f1410a5 Rotate blockchain cache (#1709) 2016-07-25 18:17:45 +02:00
Arkadiy Paronyan
f048839a4b Fixing warnings (#1705) 2016-07-25 10:21:02 +02:00
Arkadiy Paronyan
9a8fdeead9 Enable WAL and disable DB repair (#1696)
* Enable WAL

* Disable rewind
2016-07-23 17:05:34 +02:00
Tomasz Drwięga
038862fa9d Fixing clippy warnings (#1660) 2016-07-19 09:25:51 +02:00
Arkadiy Paronyan
4e447ccc68 More performance optimizations (#1649)
* Use tree index for DB

* Set uncles_hash, tx_root, receipts_root from verified block

* Use Filth instead of a bool

* Fix empty root check

* Flush block queue properly

* Expunge deref
2016-07-19 09:23:53 +02:00
Arkadiy Paronyan
dd17c766b8 Blockchain repair on missing state root (#1646)
* BC locking; Recovery batch

* Missing state root recovery

* Test
2016-07-17 23:03:29 +02:00
Arkadiy Paronyan
5ab18d1313 Performance optimizations (#1642)
* Optimize ethash verification

* disable WAL for puts

* Clear account cache after commit

* Commit only modified accounts

* Optimize existing block check

* Cache last hashes
2016-07-17 09:18:15 +02:00
Robert Habermeier
d5d0cb0b47 don't batch best block for branches (#1623) 2016-07-14 19:16:01 +02:00
Nikolay Volf
50c43bd819 Merge pull request #1610 from ethcore/db-repair
Fix bloomchain on blockchain repair
2016-07-14 01:14:57 +02:00
Robert Habermeier
36d3d0d7d7 replace synchronization primitives with those from parking_lot (#1593)
* parking_lot in cargo.toml

* replace all lock invocations with parking_lot ones

* use parking_lot synchronization primitives
2016-07-13 19:59:59 +02:00
arkpar
80d59949ff Fix bloomchain on blockchain repair 2016-07-13 14:28:46 +02:00
Gav Wood
3b662c285f Switch out .X().unwrap() for .unwrapped_X 2016-07-07 09:37:31 +02:00
arkpar
548b4e164e Attempt to fix blochchain DB sync 2016-07-04 18:24:14 +02:00
Nikolay Volf
7e452ab2e0 configurable cache size 2016-06-20 13:42:04 +03:00