Backports for beta 2.2.2 (#9976)

* version: bump beta to 2.2.2

* Add experimental RPCs flag (#9928)

* WiP

* Enable experimental RPCs.

* Keep existing blocks when restoring a Snapshot (#8643)

* Rename db_restore => client

* First step: make it compile!

* Second step: working implementation!

* Refactoring

* Fix tests

* PR Grumbles

* PR Grumbles WIP

* Migrate ancient blocks interating backward

* Early return in block migration if snapshot is aborted

* Remove RwLock getter (PR Grumble I)

* Remove dependency on `Client`: only used Traits

* Add test for recovering aborted snapshot recovery

* Add test for migrating old blocks

* Fix build

* PR Grumble I

* PR Grumble II

* PR Grumble III

* PR Grumble IV

* PR Grumble V

* PR Grumble VI

* Fix one test

* Fix test

* PR Grumble

* PR Grumbles

* PR Grumbles II

* Fix tests

* Release RwLock earlier

* Revert Cargo.lock

* Update _update ancient block_ logic: set local in `commit`

* Update typo in ethcore/src/snapshot/service.rs

Co-Authored-By: ngotchac <ngotchac@gmail.com>

* Adjust requests costs for light client (#9925)

* PIP Table Cost relative to average peers instead of max peers

* Add tracing in PIP new_cost_table

* Update stat peer_count

* Use number of leeching peers for Light serve costs

* Fix test::light_params_load_share_depends_on_max_peers (wrong type)

* Remove (now) useless test

* Remove `load_share` from LightParams.Config
Prevent div. by 0

* Add LEECHER_COUNT_FACTOR

* PR Grumble: u64 to u32 for f64 casting

* Prevent u32 overflow for avg_peer_count

* Add tests for LightSync::Statistics

* Fix empty steps (#9939)

* Don't send empty step twice or empty step then block.

* Perform basic validation of locally sealed blocks.

* Don't include empty step twice.

* prevent silent errors in daemon mode, closes #9367 (#9946)

* Fix a deadlock (#9952)

* Update informant:
  - decimal in Mgas/s
  - print every 5s (not randomly between 5s and 10s)

* Fix dead-lock in `blockchain.rs`

* Update locks ordering

* Fix light client informant while syncing (#9932)

* Add `is_idle` to LightSync to check importing status

* Use SyncStateWrapper to make sure is_idle gets updates

* Update is_major_import to use verified queue size as well

* Add comment for `is_idle`

* Add Debug to `SyncStateWrapper`

* `fn get` -> `fn into_inner`

*  ci: rearrange pipeline by logic (#9970)

* ci: rearrange pipeline by logic

* ci: rename docs script

* fix docker build (#9971)

* Deny unknown fields for chainspec (#9972)

* Add deny_unknown_fields to chainspec

* Add tests and fix existing one

* Remove serde_ignored dependency for chainspec

* Fix rpc test eth chain spec

* Fix starting_nonce_test spec

* Improve block and transaction propagation (#9954)

* Refactor sync to add priority tasks.

* Send priority tasks notifications.

* Propagate blocks, optimize transactions.

* Implement transaction propagation. Use sync_channel.

* Tone down info.

* Prevent deadlock by not waiting forever for sync lock.

* Fix lock order.

* Don't use sync_channel to prevent deadlocks.

* Fix tests.

* Fix unstable peers and slowness in sync (#9967)

* Don't sync all peers after each response

* Update formating

* Fix tests: add `continue_sync` to `Sync_step`

* Update ethcore/sync/src/chain/mod.rs

Co-Authored-By: ngotchac <ngotchac@gmail.com>

* fix rpc middlewares

* fix Cargo.lock

* json: resolve merge in spec

* rpc: fix starting_nonce_test

* ci: allow nightl job to fail

.gitlab-ci.yml

@@ -9,16 +9,8 @@ image:                             parity/rust:gitlab-ci
 variables:
   CI_SERVER_NAME:                  "GitLab CI"
   CARGO_HOME:                      "${CI_PROJECT_DIR}/.cargo"
-  BUILD_TARGET:                    ubuntu
-  BUILD_ARCH:                      amd64
   CARGO_TARGET:                    x86_64-unknown-linux-gnu
 
-cache:
-  key:                             "${CI_JOB_NAME}"
-  paths:
-    - ./target
-    - ./.cargo
-
 .releaseable_branches:             # list of git refs for building GitLab artifacts (think "pre-release binaries")
   only:                            &releaseable_branches
     - stable
@@ -52,6 +44,13 @@ test-linux:
   tags:
     - rust-stable
 
+test-audit:
+  stage:                           test
+  script:
+    - scripts/gitlab/cargo-audit.sh
+  tags:
+    - rust-stable
+
 build-linux:
   stage:                           build
   only:                            *releaseable_branches
@@ -112,56 +111,28 @@ publish-awss3:
   tags:
     - shell
 
-docs-jsonrpc:
-  stage:                            optional
+publish-docs:
+  stage:                            publish
   only:
     - tags
   except:
     - nightly
   cache: {}
   script:
-    - scripts/gitlab/docs-jsonrpc.sh
+    - scripts/gitlab/publish-docs.sh
   tags:
     - shell
 
-cargo-audit:
-  stage:                           optional
-  script:
-    - scripts/gitlab/cargo-audit.sh
-  tags:
-    - rust-stable
-
-test-android:
+build-android:
   stage:                           optional
   image:                           parity/rust-android:gitlab-ci
   variables:
     CARGO_TARGET:                  armv7-linux-androideabi
   script:
-    - scripts/gitlab/test-all.sh stable
+    - scripts/gitlab/build-unix.sh
   tags:
     - rust-arm
-
-test-darwin:
-  stage:                           optional
-  variables:
-    CARGO_TARGET:                  x86_64-apple-darwin
-    CC:                            gcc
-    CXX:                           g++
-    RUN_TESTS:                     cargo
-  script:
-    - scripts/gitlab/test-all.sh stable
-  tags:
-    - rust-osx
-
-test-windows:
-  stage:                           optional
-  variables:
-    CARGO_TARGET:                  x86_64-pc-windows-msvc
-    RUN_TESTS:                     cargo
-  script:
-    - sh scripts/gitlab/test-all.sh stable
-  tags:
-   - rust-windows
+  allow_failure:                   true
 
 test-beta:
   stage:                           optional
@@ -171,6 +142,7 @@ test-beta:
     - scripts/gitlab/test-all.sh beta
   tags:
     - rust-beta
+  allow_failure:                   true
 
 test-nightly:
   stage:                           optional
@@ -180,3 +152,4 @@ test-nightly:
     - scripts/gitlab/test-all.sh nightly
   tags:
     - rust-nightly
+  allow_failure:                   true

Cargo.lock

@@ -205,7 +205,6 @@ name = "chainspec"
 version = "0.1.0"
 dependencies = [
  "ethjson 0.1.0",
- "serde_ignored 0.0.4 (registry+https://github.com/rust-lang/crates.io-index)",
  "serde_json 1.0.32 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
@@ -424,6 +423,29 @@ name = "edit-distance"
 version = "2.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 
+[[package]]
+name = "eip712"
+version = "0.1.0"
+dependencies = [
+ "ethabi 6.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "ethereum-types 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "failure 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
+ "failure_derive 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
+ "indexmap 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)",
+ "itertools 0.7.8 (registry+https://github.com/rust-lang/crates.io-index)",
+ "keccak-hash 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
+ "lazy_static 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "lunarity-lexer 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "regex 1.0.5 (registry+https://github.com/rust-lang/crates.io-index)",
+ "rustc-hex 2.0.1 (registry+https://github.com/rust-lang/crates.io-index)",
+ "serde 1.0.80 (registry+https://github.com/rust-lang/crates.io-index)",
+ "serde_derive 1.0.80 (registry+https://github.com/rust-lang/crates.io-index)",
+ "serde_json 1.0.32 (registry+https://github.com/rust-lang/crates.io-index)",
+ "toolshed 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "validator 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "validator_derive 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "either"
 version = "1.5.0"
@@ -437,6 +459,15 @@ dependencies = [
  "heapsize 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
+[[package]]
+name = "env_logger"
+version = "0.4.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
+ "regex 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "env_logger"
 version = "0.5.13"
@@ -460,10 +491,11 @@ dependencies = [
 [[package]]
 name = "eth-secp256k1"
 version = "0.5.7"
-source = "git+https://github.com/paritytech/rust-secp256k1#db81cfea59014b4d176f10f86ed52e1a130b6822"
+source = "git+https://github.com/paritytech/rust-secp256k1#ccc06e7480148b723eb44ac56cf4d20eec380b6f"
 dependencies = [
  "arrayvec 0.4.7 (registry+https://github.com/rust-lang/crates.io-index)",
  "cc 1.0.25 (registry+https://github.com/rust-lang/crates.io-index)",
+ "cfg-if 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)",
  "rand 0.4.3 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
@@ -535,6 +567,7 @@ dependencies = [
  "byteorder 1.2.6 (registry+https://github.com/rust-lang/crates.io-index)",
  "common-types 0.1.0",
  "crossbeam 0.3.2 (registry+https://github.com/rust-lang/crates.io-index)",
+ "env_logger 0.4.3 (registry+https://github.com/rust-lang/crates.io-index)",
  "error-chain 0.12.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "ethabi 6.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "ethabi-contract 6.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -1061,6 +1094,26 @@ dependencies = [
  "vm 0.1.0",
 ]
 
+[[package]]
+name = "failure"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "backtrace 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
+ "failure_derive 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "failure_derive"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "proc-macro2 0.4.20 (registry+https://github.com/rust-lang/crates.io-index)",
+ "quote 0.6.8 (registry+https://github.com/rust-lang/crates.io-index)",
+ "syn 0.15.11 (registry+https://github.com/rust-lang/crates.io-index)",
+ "synstructure 0.10.1 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "fake-fetch"
 version = "0.0.1"
@@ -1173,6 +1226,14 @@ dependencies = [
  "num_cpus 1.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
+[[package]]
+name = "fxhash"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "byteorder 1.2.6 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "gcc"
 version = "0.3.55"
@@ -1216,7 +1277,7 @@ dependencies = [
  "fnv 1.0.6 (registry+https://github.com/rust-lang/crates.io-index)",
  "futures 0.1.25 (registry+https://github.com/rust-lang/crates.io-index)",
  "http 0.1.13 (registry+https://github.com/rust-lang/crates.io-index)",
- "indexmap 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)",
+ "indexmap 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)",
  "log 0.4.5 (registry+https://github.com/rust-lang/crates.io-index)",
  "slab 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
  "string 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -1389,6 +1450,11 @@ dependencies = [
  "unicode-normalization 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
+[[package]]
+name = "if_chain"
+version = "0.1.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+
 [[package]]
 name = "igd"
 version = "0.7.0"
@@ -1407,7 +1473,7 @@ dependencies = [
 
 [[package]]
 name = "indexmap"
-version = "1.0.1"
+version = "1.0.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 
 [[package]]
@@ -1442,6 +1508,14 @@ dependencies = [
  "either 1.5.0 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
+[[package]]
+name = "itertools"
+version = "0.7.8"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "either 1.5.0 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "itoa"
 version = "0.4.3"
@@ -1488,7 +1562,7 @@ dependencies = [
 [[package]]
 name = "jsonrpc-core"
 version = "9.0.0"
-source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#207a277b098943864ecaf22dbab7a5e309866d6b"
+source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#f8a54f46f7f1d68b4e7899ca1e929803bf966a5b"
 dependencies = [
  "futures 0.1.25 (registry+https://github.com/rust-lang/crates.io-index)",
  "log 0.4.5 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -1500,7 +1574,7 @@ dependencies = [
 [[package]]
 name = "jsonrpc-http-server"
 version = "9.0.0"
-source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#207a277b098943864ecaf22dbab7a5e309866d6b"
+source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#f8a54f46f7f1d68b4e7899ca1e929803bf966a5b"
 dependencies = [
  "hyper 0.12.11 (registry+https://github.com/rust-lang/crates.io-index)",
  "jsonrpc-core 9.0.0 (git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2)",
@@ -1513,7 +1587,7 @@ dependencies = [
 [[package]]
 name = "jsonrpc-ipc-server"
 version = "9.0.0"
-source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#207a277b098943864ecaf22dbab7a5e309866d6b"
+source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#f8a54f46f7f1d68b4e7899ca1e929803bf966a5b"
 dependencies = [
  "jsonrpc-core 9.0.0 (git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2)",
  "jsonrpc-server-utils 9.0.0 (git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2)",
@@ -1526,7 +1600,7 @@ dependencies = [
 [[package]]
 name = "jsonrpc-macros"
 version = "9.0.0"
-source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#207a277b098943864ecaf22dbab7a5e309866d6b"
+source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#f8a54f46f7f1d68b4e7899ca1e929803bf966a5b"
 dependencies = [
  "jsonrpc-core 9.0.0 (git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2)",
  "jsonrpc-pubsub 9.0.0 (git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2)",
@@ -1536,7 +1610,7 @@ dependencies = [
 [[package]]
 name = "jsonrpc-pubsub"
 version = "9.0.0"
-source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#207a277b098943864ecaf22dbab7a5e309866d6b"
+source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#f8a54f46f7f1d68b4e7899ca1e929803bf966a5b"
 dependencies = [
  "jsonrpc-core 9.0.0 (git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2)",
  "log 0.4.5 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -1546,7 +1620,7 @@ dependencies = [
 [[package]]
 name = "jsonrpc-server-utils"
 version = "9.0.0"
-source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#207a277b098943864ecaf22dbab7a5e309866d6b"
+source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#f8a54f46f7f1d68b4e7899ca1e929803bf966a5b"
 dependencies = [
  "bytes 0.4.10 (registry+https://github.com/rust-lang/crates.io-index)",
  "globset 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -1562,7 +1636,7 @@ dependencies = [
 [[package]]
 name = "jsonrpc-tcp-server"
 version = "9.0.0"
-source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#207a277b098943864ecaf22dbab7a5e309866d6b"
+source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#f8a54f46f7f1d68b4e7899ca1e929803bf966a5b"
 dependencies = [
  "jsonrpc-core 9.0.0 (git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2)",
  "jsonrpc-server-utils 9.0.0 (git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2)",
@@ -1574,7 +1648,7 @@ dependencies = [
 [[package]]
 name = "jsonrpc-ws-server"
 version = "9.0.0"
-source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#207a277b098943864ecaf22dbab7a5e309866d6b"
+source = "git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2#f8a54f46f7f1d68b4e7899ca1e929803bf966a5b"
 dependencies = [
  "error-chain 0.12.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "jsonrpc-core 9.0.0 (git+https://github.com/paritytech/jsonrpc.git?branch=parity-2.2)",
@@ -1751,6 +1825,14 @@ dependencies = [
  "linked-hash-map 0.4.2 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
+[[package]]
+name = "lunarity-lexer"
+version = "0.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "toolshed 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "macros"
 version = "0.1.0"
@@ -2080,7 +2162,7 @@ version = "1.12.0"
 dependencies = [
  "jni 0.10.2 (registry+https://github.com/rust-lang/crates.io-index)",
  "panic_hook 0.1.0",
- "parity-ethereum 2.2.0",
+ "parity-ethereum 2.2.2",
 ]
 
 [[package]]
@@ -2096,7 +2178,7 @@ dependencies = [
 
 [[package]]
 name = "parity-ethereum"
-version = "2.2.0"
+version = "2.2.2"
 dependencies = [
  "ansi_term 0.10.2 (registry+https://github.com/rust-lang/crates.io-index)",
  "atty 0.2.11 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -2145,7 +2227,7 @@ dependencies = [
  "parity-rpc-client 1.4.0",
  "parity-runtime 0.1.0",
  "parity-updater 1.12.0",
- "parity-version 2.2.0",
+ "parity-version 2.2.2",
  "parity-whisper 0.1.0",
  "parking_lot 0.6.4 (registry+https://github.com/rust-lang/crates.io-index)",
  "pretty_assertions 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -2261,6 +2343,7 @@ version = "1.12.0"
 dependencies = [
  "ansi_term 0.10.2 (registry+https://github.com/rust-lang/crates.io-index)",
  "cid 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "eip712 0.1.0",
  "ethash 1.12.0",
  "ethcore 1.12.0",
  "ethcore-devtools 1.12.0",
@@ -2299,7 +2382,7 @@ dependencies = [
  "parity-crypto 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "parity-runtime 0.1.0",
  "parity-updater 1.12.0",
- "parity-version 2.2.0",
+ "parity-version 2.2.2",
  "parking_lot 0.6.4 (registry+https://github.com/rust-lang/crates.io-index)",
  "patricia-trie 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "pretty_assertions 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -2396,7 +2479,7 @@ dependencies = [
  "parity-bytes 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "parity-hash-fetch 1.12.0",
  "parity-path 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
- "parity-version 2.2.0",
+ "parity-version 2.2.2",
  "parking_lot 0.6.4 (registry+https://github.com/rust-lang/crates.io-index)",
  "rand 0.4.3 (registry+https://github.com/rust-lang/crates.io-index)",
  "semver 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -2406,7 +2489,7 @@ dependencies = [
 
 [[package]]
 name = "parity-version"
-version = "2.2.0"
+version = "2.2.2"
 dependencies = [
  "parity-bytes 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
  "rlp 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
@@ -3034,14 +3117,6 @@ dependencies = [
  "syn 0.15.11 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
-[[package]]
-name = "serde_ignored"
-version = "0.0.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-dependencies = [
- "serde 1.0.80 (registry+https://github.com/rust-lang/crates.io-index)",
-]
-
 [[package]]
 name = "serde_json"
 version = "1.0.32"
@@ -3176,6 +3251,17 @@ dependencies = [
  "unicode-xid 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
+[[package]]
+name = "synstructure"
+version = "0.10.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "proc-macro2 0.4.20 (registry+https://github.com/rust-lang/crates.io-index)",
+ "quote 0.6.8 (registry+https://github.com/rust-lang/crates.io-index)",
+ "syn 0.15.11 (registry+https://github.com/rust-lang/crates.io-index)",
+ "unicode-xid 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "target_info"
 version = "0.1.0"
@@ -3515,6 +3601,14 @@ dependencies = [
  "serde 1.0.80 (registry+https://github.com/rust-lang/crates.io-index)",
 ]
 
+[[package]]
+name = "toolshed"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "fxhash 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "trace-time"
 version = "0.1.1"
@@ -3687,6 +3781,34 @@ name = "utf8-ranges"
 version = "1.0.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 
+[[package]]
+name = "validator"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "idna 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)",
+ "lazy_static 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "regex 1.0.5 (registry+https://github.com/rust-lang/crates.io-index)",
+ "serde 1.0.80 (registry+https://github.com/rust-lang/crates.io-index)",
+ "serde_derive 1.0.80 (registry+https://github.com/rust-lang/crates.io-index)",
+ "serde_json 1.0.32 (registry+https://github.com/rust-lang/crates.io-index)",
+ "url 1.7.1 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
+[[package]]
+name = "validator_derive"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+dependencies = [
+ "if_chain 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
+ "lazy_static 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
+ "proc-macro2 0.4.20 (registry+https://github.com/rust-lang/crates.io-index)",
+ "quote 0.6.8 (registry+https://github.com/rust-lang/crates.io-index)",
+ "regex 1.0.5 (registry+https://github.com/rust-lang/crates.io-index)",
+ "syn 0.15.11 (registry+https://github.com/rust-lang/crates.io-index)",
+ "validator 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
+]
+
 [[package]]
 name = "vec_map"
 version = "0.8.1"
@@ -3963,6 +4085,7 @@ dependencies = [
 "checksum edit-distance 2.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "3bd26878c3d921f89797a4e1a1711919f999a9f6946bb6f5a4ffda126d297b7e"
 "checksum either 1.5.0 (registry+https://github.com/rust-lang/crates.io-index)" = "3be565ca5c557d7f59e7cfcf1844f9e3033650c929c6566f511e8005f205c1d0"
 "checksum elastic-array 0.10.0 (registry+https://github.com/rust-lang/crates.io-index)" = "88d4851b005ef16de812ea9acdb7bece2f0a40dd86c07b85631d7dafa54537bb"
+"checksum env_logger 0.4.3 (registry+https://github.com/rust-lang/crates.io-index)" = "3ddf21e73e016298f5cb37d6ef8e8da8e39f91f9ec8b0df44b7deb16a9f8cd5b"
 "checksum env_logger 0.5.13 (registry+https://github.com/rust-lang/crates.io-index)" = "15b0a4d2e39f8420210be8b27eeda28029729e2fd4291019455016c348240c38"
 "checksum error-chain 0.12.0 (registry+https://github.com/rust-lang/crates.io-index)" = "07e791d3be96241c77c43846b665ef1384606da2cd2a48730abe606a12906e02"
 "checksum eth-secp256k1 0.5.7 (git+https://github.com/paritytech/rust-secp256k1)" = "<none>"
@@ -3972,6 +4095,8 @@ dependencies = [
 "checksum ethbloom 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)" = "1a93a43ce2e9f09071449da36bfa7a1b20b950ee344b6904ff23de493b03b386"
 "checksum ethereum-types 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "35b3c5a18bc5e73a32a110ac743ec04b02bbbcd3b71d3118d40a6113d509378a"
 "checksum ethereum-types-serialize 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "4ac59a21a9ce98e188f3dace9eb67a6c4a3c67ec7fbc7218cb827852679dc002"
+"checksum failure 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "6dd377bcc1b1b7ce911967e3ec24fa19c3224394ec05b54aa7b083d498341ac7"
+"checksum failure_derive 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "64c2d913fe8ed3b6c6518eedf4538255b989945c14c2a7d5cbff62a5e2120596"
 "checksum fake-simd 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "e88a8acf291dafb59c2d96e8f59828f3838bb1a70398823ade51a84de6a6deed"
 "checksum fdlimit 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b1ee15a7050e5580b3712877157068ea713b245b080ff302ae2ca973cfcd9baa"
 "checksum fixed-hash 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "0d5ec8112f00ea8a483e04748a85522184418fd1cf02890b626d8fc28683f7de"
@@ -3982,6 +4107,7 @@ dependencies = [
 "checksum fuchsia-zircon-sys 0.3.3 (registry+https://github.com/rust-lang/crates.io-index)" = "3dcaa9ae7725d12cdb85b3ad99a434db70b468c09ded17e012d86b5c1010f7a7"
 "checksum futures 0.1.25 (registry+https://github.com/rust-lang/crates.io-index)" = "49e7653e374fe0d0c12de4250f0bdb60680b8c80eed558c5c7538eec9c89e21b"
 "checksum futures-cpupool 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)" = "ab90cde24b3319636588d0c35fe03b1333857621051837ed769faefb4c2162e4"
+"checksum fxhash 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "c31b6d751ae2c7f11320402d34e41349dd1016f8d5d45e48c4312bc8625af50c"
 "checksum gcc 0.3.55 (registry+https://github.com/rust-lang/crates.io-index)" = "8f5f3913fa0bfe7ee1fd8248b6b9f42a5af4b9d65ec2dd2c3c26132b950ecfc2"
 "checksum generic-array 0.9.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ef25c5683767570c2bbd7deba372926a55eaae9982d7726ee2a1050239d45b9d"
 "checksum getopts 0.2.18 (registry+https://github.com/rust-lang/crates.io-index)" = "0a7292d30132fb5424b354f5dc02512a86e4c516fe544bb7a25e7f266951b797"
@@ -4001,13 +4127,15 @@ dependencies = [
 "checksum hyper 0.12.11 (registry+https://github.com/rust-lang/crates.io-index)" = "78d50abbd1790e0f4c74cb1d4a2211b439bac661d54107ad5564c55e77906762"
 "checksum hyper-rustls 0.14.0 (registry+https://github.com/rust-lang/crates.io-index)" = "68f2aa6b1681795bf4da8063f718cd23145aa0c9a5143d9787b345aa60d38ee4"
 "checksum idna 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "38f09e0f0b1fb55fdee1f17470ad800da77af5186a1a76c026b679358b7e844e"
+"checksum if_chain 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "4bac95d9aa0624e7b78187d6fb8ab012b41d9f6f54b1bcb61e61c4845f8357ec"
 "checksum igd 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "8a254e265e8810deb357a9de757f784787ec415d056ededf410c0aa460afee9e"
-"checksum indexmap 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "08173ba1e906efb6538785a8844dd496f5d34f0a2d88038e95195172fc667220"
+"checksum indexmap 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7e81a7c05f79578dbc15793d8b619db9ba32b4577003ef3af1a91c416798c58d"
 "checksum integer-encoding 1.0.5 (registry+https://github.com/rust-lang/crates.io-index)" = "26746cbc2e680af687e88d717f20ff90079bd10fc984ad57d277cd0e37309fa5"
 "checksum interleaved-ordered 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "141340095b15ed7491bd3d4ced9d20cebfb826174b6bb03386381f62b01e3d77"
 "checksum iovec 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "dbe6e417e7d0975db6512b90796e8ce223145ac4e33c377e4a42882a0e88bb08"
 "checksum ipnetwork 0.12.8 (registry+https://github.com/rust-lang/crates.io-index)" = "70783119ac90828aaba91eae39db32c6c1b8838deea3637e5238efa0130801ab"
 "checksum itertools 0.5.10 (registry+https://github.com/rust-lang/crates.io-index)" = "4833d6978da405305126af4ac88569b5d71ff758581ce5a987dbfa3755f694fc"
+"checksum itertools 0.7.8 (registry+https://github.com/rust-lang/crates.io-index)" = "f58856976b776fedd95533137617a02fb25719f40e7d9b01c7043cd65474f450"
 "checksum itoa 0.4.3 (registry+https://github.com/rust-lang/crates.io-index)" = "1306f3464951f30e30d12373d31c79fbd52d236e5e896fd92f96ec7babbbe60b"
 "checksum jni 0.10.2 (registry+https://github.com/rust-lang/crates.io-index)" = "1ecfa3b81afc64d9a6539c4eece96ac9a93c551c713a313800dade8e33d7b5c1"
 "checksum jni-sys 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "8eaf4bc02d17cbdd7ff4c7438cafcdf7fb9a4613313ad11b4f8fefe7d3fa0130"
@@ -4038,6 +4166,7 @@ dependencies = [
 "checksum log 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)" = "e19e8d5c34a3e0e2223db8e060f9e8264aeeb5c5fc64a4ee9965c062211c024b"
 "checksum log 0.4.5 (registry+https://github.com/rust-lang/crates.io-index)" = "d4fcce5fa49cc693c312001daf1d13411c4a5283796bac1084299ea3e567113f"
 "checksum lru-cache 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "4d06ff7ff06f729ce5f4e227876cb88d10bc59cd4ae1e09fbb2bde15c850dc21"
+"checksum lunarity-lexer 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "8a1670671f305792567116d4660e6e5bd785d6fa973e817c3445c0a7a54cecb6"
 "checksum matches 0.1.8 (registry+https://github.com/rust-lang/crates.io-index)" = "7ffc5c5338469d4d3ea17d269fa8ea3512ad247247c30bd2df69e68309ed0a08"
 "checksum memchr 2.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "4b3629fe9fdbff6daa6c33b90f7c08355c1aca05a3d01fa8063b822fcf185f3b"
 "checksum memmap 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)" = "e2ffa2c986de11a9df78620c01eeaaf27d94d3ff02bf81bfcca953102dd0c6ff"
@@ -4138,7 +4267,6 @@ dependencies = [
 "checksum semver-parser 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "388a1df253eca08550bef6c72392cfe7c30914bf41df5269b68cbd6ff8f570a3"
 "checksum serde 1.0.80 (registry+https://github.com/rust-lang/crates.io-index)" = "15c141fc7027dd265a47c090bf864cf62b42c4d228bbcf4e51a0c9e2b0d3f7ef"
 "checksum serde_derive 1.0.80 (registry+https://github.com/rust-lang/crates.io-index)" = "225de307c6302bec3898c51ca302fc94a7a1697ef0845fcee6448f33c032249c"
-"checksum serde_ignored 0.0.4 (registry+https://github.com/rust-lang/crates.io-index)" = "190e9765dcedb56be63b6e0993a006c7e3b071a016a304736e4a315dc01fb142"
 "checksum serde_json 1.0.32 (registry+https://github.com/rust-lang/crates.io-index)" = "43344e7ce05d0d8280c5940cabb4964bea626aa58b1ec0e8c73fa2a8512a38ce"
 "checksum sha1 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "cc30b1e1e8c40c121ca33b86c23308a090d19974ef001b4bf6e61fd1a0fb095c"
 "checksum sha1 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)" = "171698ce4ec7cbb93babeb3190021b4d72e96ccb98e33d277ae4ea959d6f2d9e"
@@ -4157,6 +4285,7 @@ dependencies = [
 "checksum strsim 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)" = "b4d15c810519a91cf877e7e36e63fe068815c678181439f2f29e2562147c3694"
 "checksum strsim 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "bb4f380125926a99e52bc279241539c018323fab05ad6368b56f93d9369ff550"
 "checksum syn 0.15.11 (registry+https://github.com/rust-lang/crates.io-index)" = "b036b7b35e846707c0e55c2c9441fa47867c0f87fca416921db3261b1d8c741a"
+"checksum synstructure 0.10.1 (registry+https://github.com/rust-lang/crates.io-index)" = "73687139bf99285483c96ac0add482c3776528beac1d97d444f6e91f203a2015"
 "checksum target_info 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "c63f48baada5c52e65a29eef93ab4f8982681b67f9e8d29c7b05abcfec2b9ffe"
 "checksum tempdir 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)" = "15f2b5fb00ccdf689e0149d1b1b3c03fead81c2b37735d812fa8bddbbf41b6d8"
 "checksum term_size 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)" = "9e5b9a66db815dcfd2da92db471106457082577c3c278d4138ab3e3b4e189327"
@@ -4189,6 +4318,7 @@ dependencies = [
 "checksum tokio-udp 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "da941144b816d0dcda4db3a1ba87596e4df5e860a72b70783fe435891f80601c"
 "checksum tokio-uds 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "22e3aa6d1fcc19e635418dc0a30ab5bd65d347973d6f43f1a37bf8d9d1335fc9"
 "checksum toml 0.4.8 (registry+https://github.com/rust-lang/crates.io-index)" = "4a2ecc31b0351ea18b3fe11274b8db6e4d82bce861bbb22e6dbed40417902c65"
+"checksum toolshed 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "450441e131c7663af72e63a33c02a6a1fbaaa8601dc652ed6757813bb55aeec7"
 "checksum trace-time 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "dbe82f2f0bf1991e163e757baf044282823155dd326e70f44ce2186c3c320cc9"
 "checksum transaction-pool 1.13.3 (registry+https://github.com/rust-lang/crates.io-index)" = "e5866e5126b14358f1d7af4bf51a0be677a363799b90e655edcec8254edef1d2"
 "checksum transient-hashmap 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)" = "aeb4b191d033a35edfce392a38cdcf9790b6cebcb30fa690c312c29da4dc433e"
@@ -4211,6 +4341,8 @@ dependencies = [
 "checksum untrusted 0.6.2 (registry+https://github.com/rust-lang/crates.io-index)" = "55cd1f4b4e96b46aeb8d4855db4a7a9bd96eeeb5c6a1ab54593328761642ce2f"
 "checksum url 1.7.1 (registry+https://github.com/rust-lang/crates.io-index)" = "2a321979c09843d272956e73700d12c4e7d3d92b2ee112b31548aef0d4efc5a6"
 "checksum utf8-ranges 1.0.1 (registry+https://github.com/rust-lang/crates.io-index)" = "fd70f467df6810094968e2fce0ee1bd0e87157aceb026a8c083bcf5e25b9efe4"
+"checksum validator 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)" = "236a5eda3df2c877872e98dbc55d497d943792e6405d8fc65bd4f8a5e3b53c99"
+"checksum validator_derive 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)" = "d360d6f5754972c0c1da14fb3d5580daa31aee566e1e45e2f8d3bf5950ecd3e9"
 "checksum vec_map 0.8.1 (registry+https://github.com/rust-lang/crates.io-index)" = "05c78687fb1a80548ae3250346c3db86a80a7cdd77bda190189f2d0a0987c81a"
 "checksum vergen 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "8c3365f36c57e5df714a34be40902b27a992eeddb9996eca52d0584611cf885d"
 "checksum version_check 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "914b1a6776c4c929a602fafd8bc742e06365d4bcbe48c30f9cca5824f70dc9dd"

Cargo.toml

@@ -2,7 +2,7 @@
 description = "Parity Ethereum client"
 name = "parity-ethereum"
 # NOTE Make sure to update util/version/Cargo.toml as well
-version = "2.2.0"
+version = "2.2.2"
 license = "GPL-3.0"
 authors = ["Parity Technologies <admin@parity.io>"]
 
@@ -136,4 +136,4 @@ members = [
 	"util/keccak-hasher",
 	"util/patricia-trie-ethereum",
 	"util/fastmap",
-]
+]

chainspec/Cargo.toml

@@ -6,4 +6,3 @@ authors = ["Marek Kotewicz <marek@parity.io>"]
 [dependencies]
 ethjson = { path = "../json" }
 serde_json = "1.0"
-serde_ignored = "0.0.4"

chainspec/src/main.rs

@@ -15,10 +15,8 @@
 // along with Parity.  If not, see <http://www.gnu.org/licenses/>.
 
 extern crate serde_json;
-extern crate serde_ignored;
 extern crate ethjson;
 
-use std::collections::BTreeSet;
 use std::{fs, env, process};
 use ethjson::spec::Spec;
 
@@ -41,24 +39,11 @@ fn main() {
 		Err(_) => quit(&format!("{} could not be opened", path)),
 	};
 
-	let mut unused = BTreeSet::new();
-	let mut deserializer = serde_json::Deserializer::from_reader(file);
-
-	let spec: Result<Spec, _> = serde_ignored::deserialize(&mut deserializer, |field| {
-		unused.insert(field.to_string());
-	});
+	let spec: Result<Spec, _> = serde_json::from_reader(file);
 
 	if let Err(err) = spec {
 		quit(&format!("{} {}", path, err.to_string()));
 	}
 
-	if !unused.is_empty() {
-		let err = unused.into_iter()
-			.map(|field| format!("{} unexpected field `{}`", path, field))
-			.collect::<Vec<_>>()
-			.join("\n");
-		quit(&err);
-	}
-
 	println!("{} is valid", path);
 }

ethcore/Cargo.toml

@@ -76,6 +76,7 @@ hardware-wallet = { path = "../hw" }
 fake-hardware-wallet = { path = "../util/fake-hardware-wallet" }
 
 [dev-dependencies]
+env_logger = "0.4"
 tempdir = "0.3"
 trie-standardmap = "0.1"
 

ethcore/light/src/net/mod.rs

@@ -28,7 +28,7 @@ use parking_lot::{Mutex, RwLock};
 use provider::Provider;
 use request::{Request, NetworkRequests as Requests, Response};
 use rlp::{RlpStream, Rlp};
-use std::collections::{HashMap, HashSet};
+use std::collections::{HashMap, HashSet, VecDeque};
 use std::fmt;
 use std::ops::{BitOr, BitAnd, Not};
 use std::sync::Arc;
@@ -38,7 +38,7 @@ use std::time::{Duration, Instant};
 use self::request_credits::{Credits, FlowParams};
 use self::context::{Ctx, TickCtx};
 use self::error::Punishment;
-use self::load_timer::{LoadDistribution, NullStore};
+use self::load_timer::{LoadDistribution, NullStore, MOVING_SAMPLE_SIZE};
 use self::request_set::RequestSet;
 use self::id_guard::IdGuard;
 
@@ -70,6 +70,16 @@ const PROPAGATE_TIMEOUT_INTERVAL: Duration = Duration::from_secs(5);
 const RECALCULATE_COSTS_TIMEOUT: TimerToken = 3;
 const RECALCULATE_COSTS_INTERVAL: Duration = Duration::from_secs(60 * 60);
 
+const STATISTICS_TIMEOUT: TimerToken = 4;
+const STATISTICS_INTERVAL: Duration = Duration::from_secs(15);
+
+/// Maximum load share for the light server
+pub const MAX_LIGHTSERV_LOAD: f64 = 0.5;
+
+/// Factor to multiply leecher count to cater for
+/// extra sudden connections (should be >= 1.0)
+pub const LEECHER_COUNT_FACTOR: f64 = 1.25;
+
 // minimum interval between updates.
 const UPDATE_INTERVAL: Duration = Duration::from_millis(5000);
 
@@ -256,18 +266,18 @@ pub trait Handler: Send + Sync {
 pub struct Config {
 	/// How many stored seconds of credits peers should be able to accumulate.
 	pub max_stored_seconds: u64,
-	/// How much of the total load capacity each peer should be allowed to take.
-	pub load_share: f64,
+	/// The network config median peers (used as default peer count)
+	pub median_peers: f64,
 }
 
 impl Default for Config {
 	fn default() -> Self {
-		const LOAD_SHARE: f64 = 1.0 / 25.0;
+		const MEDIAN_PEERS: f64 = 25.0;
 		const MAX_ACCUMULATED: u64 = 60 * 5; // only charge for 5 minutes.
 
 		Config {
 			max_stored_seconds: MAX_ACCUMULATED,
-			load_share: LOAD_SHARE,
+			median_peers: MEDIAN_PEERS,
 		}
 	}
 }
@@ -335,6 +345,42 @@ mod id_guard {
 	}
 }
 
+/// Provides various statistics that could
+/// be used to compute costs
+pub struct Statistics {
+	/// Samples of peer count
+	peer_counts: VecDeque<usize>,
+}
+
+impl Statistics {
+	/// Create a new Statistics instance
+	pub fn new() -> Self {
+		Statistics {
+			peer_counts: VecDeque::with_capacity(MOVING_SAMPLE_SIZE),
+		}
+	}
+
+	/// Add a new peer_count sample
+	pub fn add_peer_count(&mut self, peer_count: usize) {
+		while self.peer_counts.len() >= MOVING_SAMPLE_SIZE {
+			self.peer_counts.pop_front();
+		}
+		self.peer_counts.push_back(peer_count);
+	}
+
+	/// Get the average peer count from previous samples. Is always >= 1.0
+	pub fn avg_peer_count(&self) -> f64 {
+		let len = self.peer_counts.len();
+		if len == 0 {
+			return 1.0;
+		}
+		let avg = self.peer_counts.iter()
+			.fold(0, |sum: u32, &v| sum.saturating_add(v as u32)) as f64
+			/ len as f64;
+		avg.max(1.0)
+	}
+}
+
 /// This is an implementation of the light ethereum network protocol, abstracted
 /// over a `Provider` of data and a p2p network.
 ///
@@ -359,6 +405,7 @@ pub struct LightProtocol {
 	req_id: AtomicUsize,
 	sample_store: Box<SampleStore>,
 	load_distribution: LoadDistribution,
+	statistics: RwLock<Statistics>,
 }
 
 impl LightProtocol {
@@ -369,9 +416,11 @@ impl LightProtocol {
 		let genesis_hash = provider.chain_info().genesis_hash;
 		let sample_store = params.sample_store.unwrap_or_else(|| Box::new(NullStore));
 		let load_distribution = LoadDistribution::load(&*sample_store);
+		// Default load share relative to median peers
+		let load_share = MAX_LIGHTSERV_LOAD / params.config.median_peers;
 		let flow_params = FlowParams::from_request_times(
 			|kind| load_distribution.expected_time(kind),
-			params.config.load_share,
+			load_share,
 			Duration::from_secs(params.config.max_stored_seconds),
 		);
 
@@ -389,6 +438,7 @@ impl LightProtocol {
 			req_id: AtomicUsize::new(0),
 			sample_store,
 			load_distribution,
+			statistics: RwLock::new(Statistics::new()),
 		}
 	}
 
@@ -408,6 +458,16 @@ impl LightProtocol {
 		)
 	}
 
+	/// Get the number of active light peers downloading from the
+	/// node
+	pub fn leecher_count(&self) -> usize {
+		let credit_limit = *self.flow_params.read().limit();
+		// Count the number of peers that used some credit
+		self.peers.read().iter()
+			.filter(|(_, p)| p.lock().local_credits.current() < credit_limit)
+			.count()
+	}
+
 	/// Make a request to a peer.
 	///
 	/// Fails on: nonexistent peer, network error, peer not server,
@@ -772,12 +832,16 @@ impl LightProtocol {
 	fn begin_new_cost_period(&self, io: &IoContext) {
 		self.load_distribution.end_period(&*self.sample_store);
 
+		let avg_peer_count = self.statistics.read().avg_peer_count();
+		// Load share relative to average peer count +LEECHER_COUNT_FACTOR%
+		let load_share = MAX_LIGHTSERV_LOAD / (avg_peer_count * LEECHER_COUNT_FACTOR);
 		let new_params = Arc::new(FlowParams::from_request_times(
 			|kind| self.load_distribution.expected_time(kind),
-			self.config.load_share,
+			load_share,
 			Duration::from_secs(self.config.max_stored_seconds),
 		));
 		*self.flow_params.write() = new_params.clone();
+		trace!(target: "pip", "New cost period: avg_peers={} ; cost_table:{:?}", avg_peer_count, new_params.cost_table());
 
 		let peers = self.peers.read();
 		let now = Instant::now();
@@ -797,6 +861,11 @@ impl LightProtocol {
 			peer_info.awaiting_acknowledge = Some((now, new_params.clone()));
 		}
 	}
+
+	fn tick_statistics(&self) {
+		let leecher_count = self.leecher_count();
+		self.statistics.write().add_peer_count(leecher_count);
+	}
 }
 
 impl LightProtocol {
@@ -1099,6 +1168,8 @@ impl NetworkProtocolHandler for LightProtocol {
 			.expect("Error registering sync timer.");
 		io.register_timer(RECALCULATE_COSTS_TIMEOUT, RECALCULATE_COSTS_INTERVAL)
 			.expect("Error registering request timer interval token.");
+		io.register_timer(STATISTICS_TIMEOUT, STATISTICS_INTERVAL)
+			.expect("Error registering statistics timer.");
 	}
 
 	fn read(&self, io: &NetworkContext, peer: &PeerId, packet_id: u8, data: &[u8]) {
@@ -1119,6 +1190,7 @@ impl NetworkProtocolHandler for LightProtocol {
 			TICK_TIMEOUT => self.tick_handlers(&io),
 			PROPAGATE_TIMEOUT => self.propagate_transactions(&io),
 			RECALCULATE_COSTS_TIMEOUT => self.begin_new_cost_period(&io),
+			STATISTICS_TIMEOUT => self.tick_statistics(),
 			_ => warn!(target: "pip", "received timeout on unknown token {}", timer),
 		}
 	}

ethcore/light/src/net/tests/mod.rs

@@ -22,9 +22,10 @@ use ethcore::client::{EachBlockWith, TestBlockChainClient};
 use ethcore::encoded;
 use ethcore::ids::BlockId;
 use ethereum_types::{H256, U256, Address};
-use net::{LightProtocol, Params, packet, Peer};
+use net::{LightProtocol, Params, packet, Peer, Statistics};
 use net::context::IoContext;
 use net::status::{Capabilities, Status};
+use net::load_timer::MOVING_SAMPLE_SIZE;
 use network::{PeerId, NodeId};
 use provider::Provider;
 use request;
@@ -780,3 +781,34 @@ fn get_transaction_index() {
 	let expected = Expect::Respond(packet::RESPONSE, response);
 	proto.handle_packet(&expected, 1, packet::REQUEST, &request_body);
 }
+
+#[test]
+fn sync_statistics() {
+	let mut stats = Statistics::new();
+
+	// Empty set should return 1.0
+	assert_eq!(stats.avg_peer_count(), 1.0);
+
+	// Average < 1.0 should return 1.0
+	stats.add_peer_count(0);
+	assert_eq!(stats.avg_peer_count(), 1.0);
+
+	stats = Statistics::new();
+
+	const N: f64 = 50.0;
+
+	for i in 1..(N as usize + 1) {
+		stats.add_peer_count(i);
+	}
+
+	// Compute the average for the sum 1..N
+	assert_eq!(stats.avg_peer_count(), N * (N + 1.0) / 2.0 / N);
+
+	for _ in 1..(MOVING_SAMPLE_SIZE + 1) {
+		stats.add_peer_count(40);
+	}
+
+	// Test that it returns the average of the last
+	// `MOVING_SAMPLE_SIZE` values
+	assert_eq!(stats.avg_peer_count(), 40.0);
+}

ethcore/light/src/provider.rs

@@ -176,8 +176,8 @@ impl<T: ProvingBlockChainClient + ?Sized> Provider for T {
 	}
 
 	fn block_receipts(&self, req: request::CompleteReceiptsRequest) -> Option<request::ReceiptsResponse> {
-		BlockChainClient::encoded_block_receipts(self, &req.hash)
-			.map(|x| ::request::ReceiptsResponse { receipts: ::rlp::decode_list(&x) })
+		BlockChainClient::block_receipts(self, &req.hash)
+			.map(|x| ::request::ReceiptsResponse { receipts: x.receipts })
 	}
 
 	fn account_proof(&self, req: request::CompleteAccountRequest) -> Option<request::AccountResponse> {

ethcore/res/ethereum/classic.json

@@ -3068,17 +3068,18 @@
 		]
 	  },
 	"nodes": [
-		"enode://e809c4a2fec7daed400e5e28564e23693b23b2cc5a019b612505631bbe7b9ccf709c1796d2a3d29ef2b045f210caf51e3c4f5b6d3587d43ad5d6397526fa6179@174.112.32.157:30303",
-		"enode://6e538e7c1280f0a31ff08b382db5302480f775480b8e68f8febca0ceff81e4b19153c6f8bf60313b93bef2cc34d34e1df41317de0ce613a201d1660a788a03e2@52.206.67.235:30303",
+		"enode://efd48ad0879eeb7f9cb5e50f33f7bc21e805a72e90361f145baaa22dd75d111e7cd9c93f1b7060dcb30aa1b3e620269336dbf32339fea4c18925a4c15fe642df@18.205.66.229:30303",
 		"enode://5fbfb426fbb46f8b8c1bd3dd140f5b511da558cd37d60844b525909ab82e13a25ee722293c829e52cb65c2305b1637fa9a2ea4d6634a224d5f400bfe244ac0de@162.243.55.45:30303",
-		"enode://42d8f29d1db5f4b2947cd5c3d76c6d0d3697e6b9b3430c3d41e46b4bb77655433aeedc25d4b4ea9d8214b6a43008ba67199374a9b53633301bca0cd20c6928ab@104.155.176.151:30303",
-		"enode://814920f1ec9510aa9ea1c8f79d8b6e6a462045f09caa2ae4055b0f34f7416fca6facd3dd45f1cf1673c0209e0503f02776b8ff94020e98b6679a0dc561b4eba0@104.154.136.117:30303",
-		"enode://72e445f4e89c0f476d404bc40478b0df83a5b500d2d2e850e08eb1af0cd464ab86db6160d0fde64bd77d5f0d33507ae19035671b3c74fec126d6e28787669740@104.198.71.200:30303",
-		"enode://39abab9d2a41f53298c0c9dc6bbca57b0840c3ba9dccf42aa27316addc1b7e56ade32a0a9f7f52d6c5db4fe74d8824bcedfeaecf1a4e533cacb71cf8100a9442@144.76.238.49:30303",
-		"enode://f50e675a34f471af2438b921914b5f06499c7438f3146f6b8936f1faeb50b8a91d0d0c24fb05a66f05865cd58c24da3e664d0def806172ddd0d4c5bdbf37747e@144.76.238.49:30306",
-		"enode://83b33409349ffa25e150555f7b4f8deebc68f3d34d782129dc3c8ba07b880c209310a4191e1725f2f6bef59bce9452d821111eaa786deab08a7e6551fca41f4f@159.89.223.6:30303",
-		"enode://5cd218959f8263bc3721d7789070806b0adff1a0ed3f95ec886fb469f9362c7507e3b32b256550b9a7964a23a938e8d42d45a0c34b332bfebc54b29081e83b93@35.187.57.94:30303",
-		"enode://6dd3ac8147fa82e46837ec8c3223d69ac24bcdbab04b036a3705c14f3a02e968f7f1adfcdb002aacec2db46e625c04bf8b5a1f85bb2d40a479b3cc9d45a444af@104.237.131.102:30303"
+		"enode://6dd3ac8147fa82e46837ec8c3223d69ac24bcdbab04b036a3705c14f3a02e968f7f1adfcdb002aacec2db46e625c04bf8b5a1f85bb2d40a479b3cc9d45a444af@104.237.131.102:30303",
+		"enode://b9e893ea9cb4537f4fed154233005ae61b441cd0ecd980136138c304fefac194c25a16b73dac05fc66a4198d0c15dd0f33af99b411882c68a019dfa6bb703b9d@18.130.93.66:30303",
+		"enode://3fe9705a02487baea45c1ffebfa4d84819f5f1e68a0dbc18031553242a6a08e39499b61e361a52c2a92f9553efd63763f6fdd34692be0d4ba6823bb2fc346009@178.62.238.75:30303",
+		"enode://d50facc65e46bda6ff594b6e95491efa16e067de41ae96571d9f3cb853d538c44864496fa5e4df10115f02bbbaf47853f932e110a44c89227da7c30e96840596@188.166.163.187:30303",
+		"enode://a0d5c589dc02d008fe4237da9877a5f1daedee0227ab612677eabe323520f003eb5e311af335de9f7964c2092bbc2b3b7ab1cce5a074d8346959f0868b4e366e@46.101.78.44:30303",
+		"enode://c071d96b0c0f13006feae3977fb1b3c2f62caedf643df9a3655bc1b60f777f05e69a4e58bf3547bb299210092764c56df1e08380e91265baa845dca8bc0a71da@68.183.99.5:30303",
+		"enode://83b33409349ffa25e150555f7b4f8deebc68f3d34d782129dc3c8ba07b880c209310a4191e1725f2f6bef59bce9452d821111eaa786deab08a7e6551fca41f4f@206.189.68.191:30303",
+		"enode://0daae2a30f2c73b0b257746587136efb8e3479496f7ea1e943eeb9a663b72dd04582f699f7010ee02c57fc45d1f09568c20a9050ff937f9139e2973ddd98b87b@159.89.169.103:30303",
+		"enode://50808461dd73b3d70537e4c1e5fafd1132b3a90f998399af9205f8889987d62096d4e853813562dd43e7270a71c9d9d4e4dd73a534fdb22fbac98c389c1a7362@178.128.55.119:30303",
+		"enode://5cd218959f8263bc3721d7789070806b0adff1a0ed3f95ec886fb469f9362c7507e3b32b256550b9a7964a23a938e8d42d45a0c34b332bfebc54b29081e83b93@35.187.57.94:30303"
 	],
 	"accounts": {
 		"0000000000000000000000000000000000000001": { "builtin": { "name": "ecrecover", "pricing": { "linear": { "base": 3000, "word": 0 } } } },

ethcore/res/ethereum/constantinople_test.json

@@ -6,10 +6,7 @@
 				"minimumDifficulty": "0x020000",
 				"difficultyBoundDivisor": "0x0800",
 				"durationLimit": "0x0d",
-				"blockReward": {
-					"0": "0x29A2241AF62C0000",
-					"5": "0x1BC16D674EC80000"
-				},
+				"blockReward": "0x1BC16D674EC80000",
 				"homesteadTransition": "0x0",
 				"eip100bTransition": "0x0",
 				"difficultyBombDelays": {

ethcore/res/ethereum/foundation.json

@@ -177,8 +177,8 @@
 		"stateRoot": "0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544"
 	},
 	"hardcodedSync": 	{
-		"header": "f90206a0391c42ff4f047145a6b9a14179c3cc404b31d92f30693e28cf2bba41f47f6329a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d4934794b2930b35844a230f00e51431acae96fe543a0347a0fb280d4457e60a0b96577e1dde9e00905102cfd36def5c5b31dcc2284636136ea077f739c324e35c7448b14aa02186973d3c74cc1ab081498cd0c487a604873723a0e462d76b5204d14b13d1f1b39ea048a3b637f91d42c729c367d5bbdbd0a72d70b90100008003410402005000200009400030c40490500208480008414000a40048806408000080802008204400010001800c0020080c0a00400105a9080820400900240000084012030a1504030508000200005c4404a0c3490820000010400811040004708a1006910211444040c28001a800e920d00000940c200119111a10401001008044214002002080c21081801e008a320848a204400042400898004004010028840181106210080254a081112480031000410202440092c880be3894000120050500860880000108000c0080009e0000204007212840808cb80200601024020000210280100c018540b28a1041a62400000108204084000008808040000004870bc009a1914d7f8362f801837a121d8379ee5a845bbd53ca8573696e6731a0abce0f90ce69f740080eeb94d1cb13981fafe3bc6d020a44815acd86cbd3fc0a889501b04c0614e053",
-		"totalDifficulty": "7128515864594819065361",
+		"header": "f90210a0b5950d9087e0e1d3dfc18cae9388fc798d9b3932a6efb4fc073081f5561adc8da01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d4934794829bd824b016326a401d083b33d092293333a830a00caa3307575bbbe59bb8f8a89f115ccb864866c2591d8d3ec3bffa6231532e52a024278597d4fddf58a4b55fe6a8bb7bf0e6cf1c6e75158b3d0d0b1bb9bb16647da01ba199a450390854e88ea5504144be61dd681e805aed7a1cc8bea61d5b61194bb901001902100820000000020110010418b02612064240040a4051001001a0024400e2484001141a422e56073887504031028220080010040501800000080080268040050018400100cc4710800419088000005a840a201c09011040086001260040000021004842021800024204400204a8a1180a8a0002164400202000115420602005512a20410088002248c000020e801803c051202030000420810912000401010604000801543084004414c051048082100000221684005c8015a40100040170025040132500014054900482500403088004000821194124a0b20000200229b9601c230180400802020800000408e49314200190121404002080218814a08085870ac84878814c3083662001837a2147837a050f845be9e82d8fe4b883e5bda9e7a59ee4bb99e9b1bca05714f772f5e2fa3a41a8c7435407b579b7c16397cf08eca605eadd9e63db7f6a88315d0b500d2cb0a8",
+		"totalDifficulty": "7772265011609637471285",
 		"CHTs": [
 			"0x0eb474b7721727204978e92e27d31cddff56471911e424a4c8271c35f9c982cc",
 			"0xe10e94515fb5ffb7ffa9bf50db4a959b3f50c2ff75e0b8bd5f5e038749e52a11",
@@ -3346,7 +3346,108 @@
 			"0xae4a71712cc96a5b30b45e3b92c339c2e975e4ed683f4d1fcadcdc121ff7c6bf",
 			"0x226f6d8c71ec32c5eaab6b01c0fc1d00ae95e60b383d09560e90549b79eb1447",
 			"0xf3dec779841c9384df93bcefbba8700a292b570b29d286a7c9c5a442b4788a20",
-			"0x63ef48e80efa45383857adcb0f31076393260cbad058d1938345ad13faae50b4"
+			"0x63ef48e80efa45383857adcb0f31076393260cbad058d1938345ad13faae50b4",
+			"0x7ba213077278bd01ff874e760c3a0bfd4e5cf65bb18eed8b0f8076d81e16d05b",
+			"0x92eace1cff535745aae658d9ad90c372d41ce318a20716ae3b2c0958bdb48b33",
+			"0x76b2607fae8dbe532d8516c4c33e309f2c0a963bb68035f4361a2859d62d2daf",
+			"0xfb0c46a588438f4f217e183ef7772b649b5a45a318253496b01676253dedb568",
+			"0xd6dba12c700a0c5ca59b9d17f628066405de8e4399f92472cccce2e25e70ba4d",
+			"0x2390984444cbc9de301adaf0853f4f6ed2fb56fefeabce717306cf1e40da9618",
+			"0xa6992b0db82c50ce623febc7f16c8a316ce262255de11ad1914316b3d986fe66",
+			"0x96787cf8e0a70ffe5b84151ae8219362ba6d9465223c23c135590fe4cb9ab592",
+			"0x0b4e7fba24f6bed3790b8fc289e2715d9a3f4235c9316dd9d4f889f710a65ae4",
+			"0x49585a7af98ad032371df01c458fab933ac399ff4d2d35731a2d86fc136b5eb8",
+			"0x517947f22d8d56466dbb496f3e98fcd01864176af19be2112f2260e55bddbd58",
+			"0xbc0aa38cbc79c3557fa81e446071e6d531989d088c6311060bff971e430fa36b",
+			"0x5a0a88750db7ad3190dcaaf5bde2649e6fd99cc5b05979fff84ae689ed47b51b",
+			"0x9ad902d6abe59faabfa5fde485c2fe16f6f3e2c41aa8bdaba021e94a3c292dc8",
+			"0xbd6e7bf8b466940b9f619e3f433fb68c68d344974798996e454c925e2628b1b2",
+			"0x112b5e9d438eb79aa9d7b4488b6aa9f7edf7599dce805d8781510302da45e9a2",
+			"0x881255ae2cab543ee6b429e1b201c197cb35051d53f881cb51fc58f9c760e51d",
+			"0xcf56a4ed6b771e4905e6ba888390526eba8ba3483ead7585dee00c0eaea58cbc",
+			"0x12705d1609ee34bce0c82bab9da730452db57863371f5b84de84c4c135537237",
+			"0xa9abc081a75bfcde86ea830483be240cb3bcfa72ae34a16dfc35cb964dd7a457",
+			"0x65cf0c91f8567d8567ac3f00ddc19da0b916179266c7e42768fd5d7271e91092",
+			"0x07f15ccc91a47edfe9b67a79d05ed1da1410e3ac16e4a4c2147c3ef0de26ae62",
+			"0xb44eb7db9e4dbe8901bfac43ad54ef27576a3ac0c78218138715e03b3f9a782e",
+			"0x9ef66d20aeccdea66ade0fe8ccc0a85dbe9498305509c8919b670ae8e854a450",
+			"0xb10a7dbd4857f910ae8cca5d1b7a85923adaad65c4b00d1255813efef00c646f",
+			"0x20410bc64b58a1c221f7b38f293700591692c73a785d94a5548b336bd5db10da",
+			"0xed7176f398eb75ae6c1d4348d3882aade99573f8369ef00e809b1ff91b5a9191",
+			"0x35f06785e1f7e362682e969586322b1588624f6025692ca87db21cb199c382f4",
+			"0x9a6b06141660e2bd106f6d46afd2f3e64d72711ffe6a9c495887bc3932f7934e",
+			"0xd64e602a976e86d3948d0f4cf0b8f13a63558d5bdea5affb36eeb91007a653b3",
+			"0x71050553ca1da7234764a5486e4647fdad00652560eb47102c1d0fc039b4b63e",
+			"0x53ba43f79de58a957e8ac0d998ecb4999c68f423b1769d9e8df2405259dcdaa6",
+			"0x17b33d269609bd90fac57eaeb298bf489a3b8a985cfd9998d297c7e0628bc84c",
+			"0x9b47bba7ff6b283e76100e36246c0dcad836ff29a674fbb1c6d2f58e38e29f92",
+			"0x262f8387628500fa26e2cdca6b5938262c8fa81888daed41a09d880df136167f",
+			"0xb6c6d0ed2824a9818fa9476c99a810cf33a0c1fd3f2fb6b7a44603f2442525e7",
+			"0x6f3437e677dcc60cb587f6100eaa2e7ef5c76b57023b63ae61a6a8ed2750512d",
+			"0x4bb5aad759b0c6390cf4dbc6c32e78f2721c2be3f40ca3d11de880fbe90e456a",
+			"0xfbeced3e048769a07bfd50010de59cd24f2c534234abc1ee7f3fe9d7e3d9f482",
+			"0xbe871fdd431974370e8156fb9347377d1bb07990ff60296abbaaf2c38b4059d0",
+			"0xcbdf566fceb5adc33608a847c77b6f3501cdc4182ba98e04e1f7e8b8ae66ba5b",
+			"0xa06a5de4aeafdb5981ae02004a38c524116ebe557dda83abdf03141e4dfbd4cd",
+			"0x73660bb361b94287b85b6c41394bea5447e8e3ccffaba1772a765a8fc0fa9b27",
+			"0xf80439908d6173fe4fea42aeb07ddea2c644e581373dbe9dbb9d092f12fd2125",
+			"0x32d935f078bdbb9ae693bef4ab8316e7da701edf42f156568b639193b0ea24bf",
+			"0x8ef0b9f247b5b9d9e5a813877d03a518d1f06900e2fdf5743feda3537ab3777d",
+			"0x59f92f14340a77b94839b6ea61389691a6fa5bb2ddc5b2241e873b066d37fc61",
+			"0x0288bdae302901b701df46263f4b12bdd6bd76436c3ae1626936e1b4bb631ca7",
+			"0x2fcfd5a54d6bbe07d4063ef64fe4b164bd8cb89a8e03cb5fabee98bfef280e43",
+			"0xd8d1169b3a9aca48f167cd43e9617d3ecc74658ea61ace251beb370d3c05f86c",
+			"0xdb2c3ef7b8a8a18ad3a8ce7cb8a6be2e1ef2cd694fb3f1b2e0732f2e27c758cb",
+			"0xfbffbf26f4756bcf1cd217686597ffe9c1282a70b279c8b01ee5076ec31b1c2c",
+			"0x2cf1404cc8482698eccde40185740753c47319b1aa30d9204fe53922feaae9b8",
+			"0x553a9e079d09d3f8251947ab37891027fc3d041246f2e43f01321fc3523b508f",
+			"0x35954f131f276933c7ef3938f35c6011806c7dd637fa249451379003d740cbaa",
+			"0x1ee7a658e8a2368391810667c35b8b0e321d381907e2312e7789a3cc7b0341aa",
+			"0x4cb3c908526bc6ccbdd39d855b3b3649e59eec4322bf48f07598d98240101d0c",
+			"0x996eb99b0cb27fb5ec07873af17933f377ed2c131f02926bab74341c475a95a9",
+			"0x68a90de2c8ea439db0417b77a23eb66e7421268335be7e4f593a52b1093681fa",
+			"0x4325afa26bf545951a73c016d866f1c71a5855ef52202b7ad0c54dc23906936d",
+			"0xaf5111b606b0e6d699bb4fb61c8a0592af7eb45dd7357b3e896ebc3e798919ad",
+			"0x4de03498b2d532a8eb01a12edea2a15b8a907d6d23ccfc380507a5d294abce99",
+			"0x5d642ba55dc046ef1b84fe594ae2e46b844cc6590d580e2591ba9c0c1c942c24",
+			"0x8477bb69d6d23ae4f45e56c372d66f42ba7417cf5107136081b5ccdd208b1545",
+			"0x924f641a7cea7cf76ad8920b9ee7c39e74304169571b028fbe40e6ed255a90f4",
+			"0x4b39dbfae8635f286284009a0213c6d267d5f48485a25f2df855c5cac5e7d6ca",
+			"0x61827bf3a120a079111665f78ba2e5ad15943492e4ffdfc708e235f44ce92842",
+			"0xdc0c0dafc340277058a6f8e03bf4e5e733e05fa638e2a8c4ebe4521960bbdc20",
+			"0x7038c52c4e92d6ed20b19481ed63d432325f94ee2a3ad4c0d5e25fafe5b32bd6",
+			"0xe1f5a1123679ce45005d53672018e16648fd44c4a2251802f295d29ed876a0db",
+			"0x173a332438a181f6cf1773b2603556d229254861534f34ad65b2170b0cf47fb5",
+			"0x22a19ab1d828b5d05057f40f5cf798e27ec068fc584968987b7bbe440f52ac84",
+			"0x15b3e3b6e1de6b8a407ad7e325c53c4bfa9e73f23c6f3acedcdf68d759f73860",
+			"0x0347a8e310cbc921daf829002ba0c44399c4dd7c930ac1fca203272949e09dc9",
+			"0x9babeec2d3003ddb15fdcb89ec7b61c0d7f795145b58f55a17810447d5afe5be",
+			"0x62c14ff2ed414ede1e3fa7fc2fccbab0b8f284b9f38127823c3dc9eb55a08b44",
+			"0xb301a7c865abe0f42d9444b5c3b775dc6ab4c99effbe8d54cec29745ff4af01d",
+			"0x3fabed16363d78f34a13ca6b99cfc592671668d4281c5eb3ab07f4b9017a4ebc",
+			"0xb14af7c7331f7b5518e04bc31c8c2b63d9d497acac405bc22e1ab2431105b867",
+			"0xb7b90e8852ae132f858cb98635fd6e1267b141de7da6773ec4092d5c83def42b",
+			"0x1a5c8bed811fc72de8146516cc4d668fc4557baf48082503a2a5cc0e473b6e08",
+			"0x0060158b427095081a18fc57c4f56290873188960dc0e4d2b16cdd57b5761100",
+			"0x2f0dd546f4982d47b6e04ce282b19be8e165b5c89b5f03c1ac8c6d8be2af2b4f",
+			"0x229d31a50f46afeda23a2535c8b011a827383c981e7e59f67622359e4b307a10",
+			"0x10da7b2e78954845b506b9672ff93d3a617f187521116d442b364a1178f10f08",
+			"0x511325c034c772ff3a61b1c8d94e3c13f19cf1e93523a8924666833bc1e5b0ba",
+			"0x3a8b0251b2f7c99774d0a7863388422723b5405fd1db035ec5e30df90306e251",
+			"0x885c3b0507dcdd6e1cec187cf3d97ee12ce101ccf63d99c1c7fc77dd57ee0439",
+			"0xa76f1fdcf546ccb62f86ec7b85bd8eb662f696d7aed2f2ed2b3c769af90897b2",
+			"0xbec073b15a35de54487b60f92615cd44dbcb5a5fa9d1bcca2ea31437b2e45756",
+			"0x662013eba4201630ccc4a9b6fb1dac36f439cc6cd2b6f295c0260713d075108e",
+			"0x81b0148f7e690088330650b01c52831d50f3d30ba2fd2074bca5513c179ad30a",
+			"0x01614cb8f2f2bd6794a20513fffe99a36eee526b7145405db15f9c2ede980093",
+			"0xd3d8664a5464c8f681e37265948c4046e373843e602bee66c8583277397db258",
+			"0xdaf61a9e7157100cce6ea5bc405926fd4190fda7354edaafad9b6e468bb24422",
+			"0xeb05144e032621f7b1792dacbdc79c044a268442a9a9284ef1b2d1d60cadb29f",
+			"0xb1916e802e8e3eca2570d1a190d317cf38cf9f6b58119b65ed0d60b9749461ad",
+			"0x17e103cb26e7c5c7678a523f99f90387490bcca00916190e7e1b38274c52a375",
+			"0x105b8126ef037364f46a44fdca4fea1ec4c099de5a05918688249fb8844dfea0",
+			"0x7ca09f536bb9f0d7b3809c086918ba4750722edfd47153a8412fc761eb4783c8",
+			"0xc988cc2d92dcb85eaff8cf2f70ca49646cc5aff6c8e10b43a0581193137d3b92"
 		]
 	},
 	"nodes": [

ethcore/res/ethereum/kovan.json

@@ -62,8 +62,8 @@
 		"gasLimit": "0x5B8D80"
 	},
 	"hardcodedSync": {
-		"header": "f90247a01865856fb6e4118598117560df31734c74cf725c8edae4db941055ac0afeb207a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d493479400e6d2b931f55a3f1701c7389d592a7778897879a054563efd593e9682943065880710af9187131127148575efc8bb51d80dfed41aa0a568a1653a6c7d559711be0b91a8e75db76c678dbdd286c75b88e4f0c0d31171a0dab32c5cbe9b9244a7af00afa7f6042a4ac923573e8f2f025b107abe1e3da999b901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000008000000000000000000000000000000000000000000000000004000000000000000000000400000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002000000000000000000000000000000000000000000090fffffffffffffffffffffffffffffffd8389b001837a120083046540845bbd803c9fde830200048f5061726974792d457468657265756d86312e32382e30826c698416ef600fb841387b51dae8bc8daa6cde190d3f44797690b4da1ce5fcfcd54bdbb2a6ee6d8c1f7649081ca28b5cd70067ee9f61e27d8184db83705102d5e1a269f2b631b4d5db01",
-		"totalDifficulty": "3020091077015059097853315484608800838133866777",
+		"header": "f90247a0434ba1ebe3bc9d9d12886f616afee0eb785dd00d78aa64502045b249c8a3cc33a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347940010f94b296a852aaac52ea6c5ac72e03afd032da0d1c1137be913218840c44bdb8f5709a59d8608b05be0dc2401402e11ff1bf2ada0eb50a864e57e19bd4c8499725a6a3841f706ff5d743e64f1d06db903aca62eaba0d4d77727f7253c109e6f048fa5b1583b6c2911ad77f7d59d8947d38a96a67729b901000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000090fffffffffffffffffffffffffffffffe838ee001837a120083013502845bea4f689fde830200088f5061726974792d457468657265756d86312e32392e30826c698416fa93dab841c80b3e34bacdc91e01f6a76c64a920a65edfea86b84e0fd59c6d8dbd7758f2dc4e3cb446de8dc07f3d8cb71375a6315f401cd1ce54c56694e4e01f7e774555e701",
+		"totalDifficulty": "3135776192732436705400032023148164213445066062",
 		"CHTs": [
 			"0xdb9557458495268ddd69409fc1f66631ed5ff9bf6c479be6eabe5d83a460acac",
 			"0xd413800c22172be6e0b7a36348c90098955991f119ddad32c5b928e8db4deb02",
@@ -4470,7 +4470,173 @@
 			"0xbe48c727fb6a32242229eaa09146c76522dcf6bed6d1c6fc1bebf86b5e4ccdb4",
 			"0x8487b971e383272df82cd812a0bf3a2026b85bc3897b4ce9ce48afa00849fe00",
 			"0x60d18b465172f59c0d71594b5273a90cb41db24a5d4c9fc37020f9d8c467a4a2",
-			"0xab4e36d9f17c748c87d89c23b667e3f4e3265e77b62dbd9c92659026f8a53d12"
+			"0xab4e36d9f17c748c87d89c23b667e3f4e3265e77b62dbd9c92659026f8a53d12",
+			"0x2c711e8eec1be3caef6e16a03dcca83dac3a565c93327c67f4e8ea9f2697d9e9",
+			"0x21d7a9a3f22a163767075aa693d82a962e4458b074bd8f3485c4ded1e47c1172",
+			"0x1509cac83867db4b888107974d5c8547dc4aa1db3b8e886291f0f9eb6bd7af58",
+			"0xa555eb6001982080479c9bdd3da9e8668bf90ca9538f16f27a45ae698ec85fb7",
+			"0x1d1f4019fa3ea7ec85ee0a411560738791f2d45fe5fe6242d07267183a852b96",
+			"0x2a8994caa5a27eb3081cc749cda5694da7ed3fbb8a1b4c67d7e92306cbd3c6ab",
+			"0x8389736ba114022ea97a6b3e755d75e74c3c370e9eb916c0f2d73a46a6f6c396",
+			"0x6a2e5d08ea64195a4153fc7bf26b7a99dcc9d31d8f58faa07510a1e87fb1ade4",
+			"0x78b66b38fd9b9cd3a5bc9c91c6f816153c2c28c1055f7ac9ad12ab61f9464850",
+			"0xb2f77834f7a88c4354763a28080d95be44dcb380d01f09292e679a6ed274b179",
+			"0x2070456b2bff0c30ae26b2435af82e153094aeea8102a75541e02a39e9ebe717",
+			"0x4e3bb2fbdc71602a62f0c423aa45398af09cf3ad24f0437027b4590e7056b882",
+			"0xe877208750e7569ce78215659ef78c1656e98b63ab6cc3e1381d7581afbea99f",
+			"0x8c9d8132d01b83cffd1a1fd9d19a3d2fb3a58a0a28602018db44f8b6dd5fe1aa",
+			"0xcece8a371b72873a660f4077f98b04d6a3cde1a150852db3a194a293e6a08b72",
+			"0xe72c8c7211acf2d67bf904cb3428c6fd69c8bd679d52deb3b2e749d5a3124b86",
+			"0x504f53693e747fbba7475dff1aad887ad0124d48cc7892ec488fda56e31e0de5",
+			"0xddb3ce7ac0c7e2d141df857e9e5c083d66f4a62ead742bf6631756a72fb643e3",
+			"0xdc1c6a45dd8308ea319671732de7578a65a025852a3cbe88aa9c5d770d662990",
+			"0xb6a9df772bdb5e7cdff569bb9d3078c081ed53e870f9171a5b0369d84fe6ce50",
+			"0x1b436a7d8c9af357c8175430e1d0d3057096c92bd8bd24fe285fa55673c68b72",
+			"0xa57ce7fed7521e26a908ead89253bc57939459b4c34fba583eececd0f9104e2a",
+			"0xb16c4f6bdce8de6657d049e2e41f91a1cef8623671eacb9ab02fff45cee7c0c3",
+			"0x8d23f3ea29767da41a2090cfceb56a65236a405031b69e614c78fb79c1847651",
+			"0x33149d4876fa24dea4e872ad1da120e01b14e8f9f8270611fced328a36df25ca",
+			"0x266c558486a85de2536e8849d49443e7d39797a6ed817ad7217633c28166bf06",
+			"0x2073a22a1f157025228f8fc82bdaff48d3cb1c9b004ac2cb867c2cbbe7cafd95",
+			"0xcf643d70f96101c99cedcc2d169b0196c564e7a8b235ae093ad25b1ba8981d01",
+			"0x1489101bebc1b60e4c0fa37a94bb946f1b51ba284c667e4b2f3270ef9d264d41",
+			"0xd2eceef739e2601390d872bdbb58f8d67864b000f7fbab07b0354c44279379e8",
+			"0xf67a02d7e7e6da127015e5137f90e30ab43c2e828e62c3f8fe68ad5a0a3011fb",
+			"0x01fe94220a07efc2bde2404c584f67e9b06e47eba05e50e71ad59710d9d6a9fd",
+			"0x509bc80d1a464116b4f234001c3f2d7aac7772cd6ec5fa6ef9606ab8ab273127",
+			"0xea497db1233691b3b86d8fc80ebffe28e6863f445a0b352a1286412a94ad2f6f",
+			"0x80074f0e2cb668b6a8e5b454db4ae5cf2482894f3ccb7648cfa26b3365a0a54b",
+			"0x56c8cd15e5e1a21c6f32f7211d18e1e56c89da688cf11b80c9e55718514eab24",
+			"0x6705989758f03a8c4ee93d44f53b4114a10337694197186b62ac2220188a6879",
+			"0x7581fbe2b0ae6e0a61d073fd1700c3111d2e3e8d9342d50800ba05d026c9edd6",
+			"0x254548091369ad8e0abbd2eb4e31e2c08c8e9dc0256fd0097e50e75831946924",
+			"0xac3119a5ba40f303a9a70ecdef9b6d4722c6adb6592a0cf52e5a312a1a0819c2",
+			"0xbdf30bc3cd852b882c5d410e6d4419e0f36ed17b761b58fe58ded980829d1cf8",
+			"0xebaeb5965563f6fa71006d8c2d285e093a167630c319cee2dfa961c4a583b60b",
+			"0x6cd6717885bd50263a97574e8a56a3ccbca88229bce595b38a454c1184ea94d5",
+			"0xc9bba3a4a72f0f08663e3af36e2bac2d4c8422b2e3c3bd29705e5b86e7c42a20",
+			"0xcc442230a4b6aed5a1eb2b8744a09aefe3cfb9e477816edfa829dfbd3f7c2bc7",
+			"0x4ee48471e3326144c7a48b15fa5f39691ad9f24b82430ca281eb961b18019e02",
+			"0x35016779dd877530f99b1d3372a0a77eff7ae831208d2bdfcd0979b838190acd",
+			"0xe3e9b2a4adc4b4e167376b33b8bf2a00bd64d52f66f4e7b291ade123e8c04402",
+			"0x5a7d1466e2538b62ceada9577079248788e912e23c4cbc52d7b82a1afbf28ec7",
+			"0xb8529c5a2b2dbfb72e70165b9c9291e7c02b43f157092a9b59f3b4cad85fb587",
+			"0x55e90fb666a950f38ae41732269cb69afca8643b75a99bce508a16e06685fc03",
+			"0x651f45996ed080d8a8dcb78550089331ad67c2df33330d1871dff956463aed3e",
+			"0x9c6b4a961663c59dde3a4f0fdf7b68b0b5f049719ba0d10d841855cffc7ee166",
+			"0x15c4d98e0ee4fa1535f83625a20bf1a491852884f8fe608eae6b9bde45986779",
+			"0xc7891cf34fcc61935a43f37bd7f23b674c785ef3b2718358fb627364f3d8b09b",
+			"0x8105fb76c9e1281f0f0d7c520dc6cd1d546d1b4b29cfa398eecd7965af57f408",
+			"0x2a141f1c4e1f1e7be5d10c444dde3d1b5fe1a69f23506d5ed1842957375a1208",
+			"0x9d2313d34a5a6873f2ae1e2684db00e3b693ba962796fe7e78d26a2c49104471",
+			"0x84d0245b1537ec1fae89fd39741d3b6427ceda3933f750befe4e2d8ec022fc1b",
+			"0x93d281ff6bfefc7f94058a0d87cb826b97728c9593ef4876b88c35d26b2163bf",
+			"0x9018a352b09cb36311d0b54bd83a33dcd5dca5ae7032f1d399fec3a0e5a9cb7a",
+			"0xf01486ea17dca0bb45c245c9ba76d20d1a2f4eb718b7d07fb5183736e5cd08c5",
+			"0xf797fec0a8cddb31ace66223e1355211e41211c4bfd33a416c8a1a86e497f630",
+			"0xf24aef69daf4e2b90b4f3bdedddbabbae86a7fd1182947da7306c8145f82cc88",
+			"0xdd14e34cd000ddb6ae6426dcd78cabbf6bfbe5b3ecd7553c4989b3891e506257",
+			"0xc01ece645c7430801ee26b20ad215e221f7de0e060fe004b450b372a6c4438a9",
+			"0x64c836653383bcb165207b80ee8f9377ad25ef1bcd6cf7ba166cca78419898ed",
+			"0x6b1cdfa195df272d7e35ccd843191d5725919c0d8b71f1abf4b950a954770503",
+			"0xebe51f94a9f445d83180c4f075d944fa2f05b5b784b7e3cb4bc1544020bd35cf",
+			"0x7007f6b2c04420d390fbdb6a9fc21edd3249c1a098f8a9383131b8be50c6f975",
+			"0x6b7e93d96f74fc2b068104a017fa5fa74836e8be5599865309361ad9305d9bb9",
+			"0xcccf2dc238c68464f3b0446e06fc48fedf202e366ace5734cdac5a7fe7c43342",
+			"0xfec77750c277cd6bb0937d25d7f598d07cbffb1117b6232e932a296f32b45ce5",
+			"0xb04e6883c11ba3f2b7f0301936c3dcb622f6f78b3d3d57e48c6dcba063038d6e",
+			"0x23fe83ea2f197cb97a3ce5f2afd75652cae993f30bc627e8b9572a0069ae088b",
+			"0x44cc6bf2cee513c826ecc0c130a32b61b0153129721ad2a70d4503b71c128247",
+			"0xb1a11be5f9b37faf1901b1405494af32084641dc02685ec33be7f6e82e360a42",
+			"0x284bdf9a6aa0ac964b0f5e72d0d47c713c919d72fa9e24f42b65d749efb6e2c3",
+			"0x96a0a0e92ed87946a322c9bca6c0ebf70e51e0e825e2ede85b975349f9bb2caa",
+			"0xd73becb6a3d9a155d46a96d17412cf015724a79b166d2961d24125d92e355947",
+			"0xd87f142746611ce081945999f42b8da49cbda03a10bf953ba043417861e27c77",
+			"0x4af010485b5385579a534693bd21ef3768c8920c5b5dc6f9ddf722cc68fd6550",
+			"0x65a2f4ec5cf3b37530fe0672418214618393db47ad1185f285b445cf5f53cf26",
+			"0x1d208df2a2785599565a40d934eb2c0cc1673c1dd43d11685d698122ecf24d6f",
+			"0x046b7211427f0027ed4b69b06cb43986d3cd53d1579a2e078f6904a06ee6587d",
+			"0x053fb3ccc35edc1a1623f8652dd8c239045b02b58ffe7d51337950ae43365990",
+			"0x0721f82b52ff87272e1b0af5f4dc16b90dfaaaf3ba56b91bef555d283c98a8ae",
+			"0x26e93e4e2d0df49a857bfcede7e6ac234d75b71617b561ce39c3db130449e4fb",
+			"0x5ed6d53c35a0aa52d9dd0a244f31be00426292c11676737c7283f567ba485dbc",
+			"0x047add17bd1601a06a32b721165ecb2afb55290e025a828696f234b95c52eae3",
+			"0x6f9e6884745cad7091646f7767a6b17ebffb6482ee4726f5c1f3b9e02d26b77f",
+			"0xab29ed5be2938fb19614da621ea9828eb338514309b3d56efc6f19b57b8f4f6a",
+			"0xd5f14043479e8e37c6545527b2e93a76763a654661af9a25adae2b37dac40672",
+			"0xc43635c1489c3cf3aa231419bfcf7bcb2455a812e4781e5947bb76cf0219d1e4",
+			"0x72117c92f8bab83ccc7fbe5fc6cd931c765238ebd67ad8c4fcf1353eb78c8df2",
+			"0x5deed7df3559ed75d7c2388a6e583067222684741143350f85d236bc5fb4075f",
+			"0x655bbc0590587c030902e32d88cc9113fcbdd5c3371820868768abaeacecb79c",
+			"0x1641c875e8e44b1fa71badcbb4f724a71e2f916d9cd212f09e9039ef41b1242f",
+			"0x40bacdc3f0b53522cdf2aefafa2bbd9c8e9f8fd25b4b33bc19a87d72a5de293d",
+			"0xc35c6cffaf93170662a6f3d4a0152c988b0f4916a33d037801f6f589a6140d08",
+			"0xd4e37125c0ca7844e048bac4e4c6069ec24b0493e4547f53679fc8df2a1e0d66",
+			"0x34c6bcf0e01dd55a1d3a82df0a2edcb2caccbc4be93fdfde32c7718276e59bbd",
+			"0x78e50d394d9d42ca8aae3f4dfccfb32fc3923073a6643c010fd49ec3897c4fe5",
+			"0xeba75c237fb213e893083c5ed0e28dde3ec1f5f1b60edfdddc17a54de9cdfaf3",
+			"0x4f35b9a67fc66492fea331378c479eba921fe0a6f0589b5ac07a44b1ea9b4995",
+			"0xbb4d09d98b27d672259b608f5075575cca49f87cce0519813c1ba4a614d369ee",
+			"0x805550ff80be2d3a543d5076961442f70dcd86e82762149c125d2ce371a32f69",
+			"0x7d81d766cc2d6c66c7bf07be0a416f2a4c0dd2bd94026270a410ea6a051dd4aa",
+			"0x30edcef101041c0e0acecb8d6c15a28c70b8894e56196cddefb5fb6b39658ceb",
+			"0xe5e8d21a00ccabb5cb50b4d04ee1dcc217fb9908e72bf0efab69d4ce7a0b7eab",
+			"0x3793f78e2e582b56fb3d24e3bd4aa69db4741e768795f9681285b0fd34ebd602",
+			"0x4a9378b4ad76df084739b2a63459ac67abf8eff7499210936fb9e6644265054a",
+			"0x2e47b62375a267e104ad2f8bfaf5c985fa7c6f4c438739810517e41141a825bd",
+			"0x73dd638affe08c104b37d53e6cb7884b8a9b4cd4afa4bbec6ef608bab4559dd9",
+			"0xc566dce73da1cc0d069a032045c44ac4ac86b145efd2cb1c363a2d403fd8d7cd",
+			"0xa1a04d193c5408d0edf85e771c6976c5dc8de08c87d906b6c9de1efa4b4fcbca",
+			"0x3245d987f205b01c54448d9a9d76bc7e4174c1021726a68f5feb6748ac373f8d",
+			"0x1f963274b936dde92281f2acd47d3685261b03845efc7d10ab8b02500c239af0",
+			"0xa1e6d69b28aea5d24c3dc301dfb0d6df8ee8fe693d1d0f419cde3b4d20d22a89",
+			"0x31b0711795c77cae9e1b62b87244a5eafa4b1764aa1620d53be7e779db060984",
+			"0xea141234dd00c54ec7912c80baca591b53a058b81fed1673317fa4ec37c2c715",
+			"0xc5902f3941d647da304e94062951b09f4d6da5bcff71517c7a35e8b49c6ac569",
+			"0x54c5677af7e31b84010f8b2941ffed5080e9b4d18b608fcbd230f163a5e04ba5",
+			"0x1a12c56542a01bc58288a58e8e526f2b2c81a5238c83bbfe4808d206b3b9d0e4",
+			"0xa398984bab81b7640674893496b2346eaa8257aad4cc8bb52474f7162fb6fa2d",
+			"0xf454a592314f1b7459d21eeb6ffe54c96ae2e25a7051dab1f8129518d3788cd2",
+			"0x12964f85bcc410fb40e89f518d749c7dc2be7546a960849dcf560eea2ec3d333",
+			"0xb176775c2ff94aedef2c099b936468986874d17eb563086215525acbee2fdbd1",
+			"0x250de6e94d60125bb680e862fcb70823d06e4f27308ea1ecea8a465d5febc860",
+			"0x731cc6817fec665290d7637c67fdbc72bbebae70a13f9346f47f3b368eb1e4a5",
+			"0x489e955cc77f16d8aa16bada320127380315eac978f0fd1033baa04911123332",
+			"0xa88623d9f89221dce46b55ca7248581b4e3ace94a7cccc9fd44a3b03412fa729",
+			"0x4a65c1808d79577b36dfb5fa74d0deb96e90d0105ee373d552685cc4f73d35d8",
+			"0x7aa8228e6f8fb98a23e1599e12ecd84c879475a23f10660e763e338a347c9002",
+			"0xa825d02023ab76478b83a2a61dcc5cf6c05ca741b960043adba3f13d1c99612a",
+			"0xd3c90e414048510d9594ae7a593e25641b6baf93f9625cc4a3caade094f02e07",
+			"0x4d631d9024a4e75111189835b4ec0007a6951d710512dd7b91a8d56da95e57b2",
+			"0x0158c7f49affa675f04aa4ebdf11e88bab8c6dc0abfecc0100d32489b0d59f1a",
+			"0x4ec96496694ef53e17b11b3122cc45cce3a4fe1a252fe9c0d379b302db1318ac",
+			"0xeec44b94d9f6c7359047d257ffd5f1f5cee87b3ebd6845dc1fe48be686a9ba69",
+			"0xe24c7b667382186196e5da605ce75a5cd9c7e4803fb2e268b4de5848737db026",
+			"0x14a551850f945f02fa2ab84c6ffe3d17930c81f5b15e94ce92342c3646b8cd9a",
+			"0x7e9a72469a52662bbcacd449dc7df8fa75e5cd1480dac3b635148a34e0f9e375",
+			"0x2dd4fd0394574c5d8c9272c42175e3a88d8501753bb25a77fa937123d0c2538d",
+			"0x6f044eb54fd2b8e51d89f6db2ee6e654b144d9e69cddf79b7899e06af5bf39f5",
+			"0x091d894513fca5d2239f6b5ce73737aa2657c90c2af196b2b893ba0b0205859c",
+			"0x78b9ca580e3eca0808e60420ead000365eeafa60c6394e055a653a7f536c8b6d",
+			"0x595cfe325aa600ef0d7de6b2eb717b33c92dfbb1e3ecb9487fe332c878e9f121",
+			"0xe1dcc22aa44a0243f67dcb0aef7e0eb1ceee269b0faa67633698f8ee29155115",
+			"0xddd3354d4167161c6b18173ae24fffac2cff12a8bcb3ddf1d964907716685cb3",
+			"0x55c02c02eb758f8c9aab60a2432d29e73b567e3e3d0148611ce4569fa074c5d9",
+			"0x5d07d5a7d45fe16a0521c55b3dd898b922c619e5b938535e2f974f11b4edfdc7",
+			"0xf75100a1a2f8d6daf0dc36569cbf579ed6e52838b163fe50b5deaad8fe7bd07d",
+			"0xb6391d028fbc88825eabf1dc4566e58112e6d3b7cfe6d387d0fa58df387df78f",
+			"0x8a58fab34ada4d30eecbad64a001015c91e446cf14e95c9d7a3d894530787b03",
+			"0xac13b36d0eb0483ab8cf21f49c78f9e32092c74c878ff520ed3a3c0080fdf5fd",
+			"0x358c1021fbdf028ec82b811cdeb504adc2611a1ab477e8a191f780592d598361",
+			"0x6a7af25eb6d089ad63712187d1414e8bd04fb6bfbac26e6086b1a3e2d66f6f7e",
+			"0x6bebf617fe66129dc8c9dadef1a74df05567ed2fc89f4e5c4dd36327d6b0a5e5",
+			"0xc658d76eb9f3fb91327c64668b08d3e9bfbbb2099bf5d9aeb6a61b8e4f893c39",
+			"0xfe68ecc66eaa625842068f8b5ad4bdf5cbce0c82ecfafa4bf48b08de49c58595",
+			"0x26eba0a168f2e5f3ffcb37955839d37432b39f2f447497854d42d802826fb1ee",
+			"0x8b4f8964062d0f00e40a41d21852a2e22abba1f4cf75f594797e368e4df21f7d",
+			"0x916993123d91b181d145e2668efed6eae3845fd76d4765df3a04d8adde8c7142",
+			"0x21b91051b5c5fbd22164a655de8485cfc2a9eff58ed09d6fbbb454560898daed",
+			"0xb52ca76065bcf43f06335977ab87d6c809dcf4e6cdfc452ac2018b77c64089a5"
 		]
 	},
 	"accounts": {

ethcore/res/ethereum/ropsten.json

@@ -62,8 +62,8 @@
 		"gasLimit": "0x1000000"
 	},
 	"hardcodedSync":{
-		"header": "f90217a00f0e017311206b97b47403eba05a16ada760a691a36f844ab8bc9082a4efedc9a067baeee20ae4f4216ab4b0c41198efd376aca23e44b1d6575c5949955547b72a946a9ecfa04e99726ec105517ac7ae1aba550bea6ca0e64d4fe9bee34d98e127f7f94f17535582d5bc6eeb6219f323b046b9a98c72b5a02d33ce5daab0436707c6d958dcf0bcd311ec7a72d7b33c20784178d5d95bc6e9a0a5b9cd4802fafaa9381ec0aa745cdb7ac953675e9df89474e5fe14fee134cf87b90100000008018000100000000000800000010000000050004000800000800014200010000001000000000001009001000000000000000000000000000000006004020600000000200001000108088002260441000020204000000000000000000280000000000200010001000000041008002000000004004c000000001000000000008000000000000800000400000201000000044001145000000000000001000a0200c04000a00010080100000020000000400002040000000000040000000040200001020000401000000800080080000400010000000200000008020020200000101000000000100400000000004400010020000200000000000001000000008453461c8683402001837a1200832c5216845bbd359199d88301080f846765746888676f312e31302e31856c696e7578a0c1da176f6642888b4369e14349ca7dc125ef7d4f5f7abad61bd7f6b95bfd46bf887d1a171a9f55dd67",
-		"totalDifficulty": "12027449412394243",
+		"header": "f90205a0cd611d63e443c91cba1dc259c71fdb96bb70a1b78639166f7aa2a09b36843f7da01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347940d85c541489b4c67df516b86898e1ed59c8d639aa039cdbfba9ee1e91d15fead004292a7df0865204ee878daab2375683cc527df2ca0ebf5b5fe0ef82be8bda42e6767ad7881269cefe650c3a899f38f5ed1c05cad8da0578f443c4b0d0bd89e42a855436258f1b19d1a018f6769c5be7aca0679ce6008b90100000000000000000000000000200000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000200000000000000000000000008000000002000000000000000000000000000000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000000000000000000000000000040000000000000000000000000000800000000000000008501856ef4de83436801837a121d830e1038845bea0191864b564f544845a08dcfeee7d79d54276d8bc35765ade21a51d277540a3d7a8e17fc26b6324487e488d70f2a90307e2ae3",
+		"totalDifficulty": "14549054591321060",
 		"CHTs": [
 			"0x614648fc0a459451850bdfe353a932b5ff824e1b568478394f78b3ed5427e37a",
 			"0x1eae561c582dbb7f4e041998e084e165d0332c915d3a6da367638a8d24f3fafc",
@@ -2116,7 +2116,112 @@
 			"0x40384a52564fae5df8c3e41827cdf584e38f3f555a54ca749b7b516619071d85",
 			"0xe52f7c17a4106594563ae7b724e5613e87d8442177f39a36b946b0e335be0e5b",
 			"0x7726202a7c255a9d7be47af6f3b44c1e379cda61237866554396fb6ec043482c",
-			"0x665da353f296cde80e3cbcb9c7f93b107d5676e5cd694b0bd33334d74789beb9"
+			"0x665da353f296cde80e3cbcb9c7f93b107d5676e5cd694b0bd33334d74789beb9",
+			"0xa1157fb181aaa945793b029d3915b37103d050f1f695862d9cda90df4755c189",
+			"0x449c9daae1c38b3d3f6861b63dc611f147b8a29029927f85c32b5e549df8ee9d",
+			"0x5739fcf908f960416e5227ff6f95aeb00696f8eb7192968239458a0aebf42533",
+			"0xfaa6017362d6e64f9ae1b6d11764ddba77cd980261acb5bdbb17b7cdee2d3024",
+			"0x97fee586909cf8b3cfb2f2dd4e251cb642eb551f1e5d9fa557a21dcb66f430c0",
+			"0x9934d90e0b4ec5900107784116323b0f76d1d71491cf9c619520478b6eb97ded",
+			"0x8f5afe4fefb0de810442e27b2bce63e3a424f9e339a0e1f46f6ed26617a4a404",
+			"0xcd12ed1b75e624f93d9e4fe7dc65407b6d7963196e4082fb459ce354977355a1",
+			"0x6858ff0c07de87aa2a88498ae948efe6cffade00f3f21086f0a71b8364090846",
+			"0xaca944e99122e4fe41c62b88e67940332483c8aae2a3a615508f848ceb1c044e",
+			"0xa8bd3a6b9197bec8849737e6771f563d14d7415707776ce4bed9460588c55c9a",
+			"0x20e5f18e795844cfcf2326f52ab1e56f4a53a1ce405fd355e0d37de5f6d552ee",
+			"0x038821cd156d2d9e2aff1ce3525726c42f581ca3e9e2cc75df524a3dbdfe0feb",
+			"0xb79bd23e10d1e6c7ae749cecaa732966ce3c2e2ea8b640d43638f14d7b78a21c",
+			"0x2c4d3bcae4a76ffe6924555b320847211367ee37899b187d8fd4358d91264c2e",
+			"0x0896db7657d2aa8d02166bd4ceba7515b24147fcf0aa34223590fb80db64f848",
+			"0x95f3b077d9cff92709d46185a7dc5d723f257fb77641e436b0f1bf07947844a1",
+			"0x6fdd931a93229698b92e6419074bc308a372e0923b35321b725916c1c151cae0",
+			"0x8c337c36c2027d0f2fb96405f76d3190cdd01237f6a602e07976e09c4a2339e4",
+			"0x8091a859cf86c1e2c22c870ebc10334f50e05bac2a41153102bf3f698bf5156b",
+			"0x382435cc836bf1873e489cb4d42d1c1673c5d1b452ba2d16dbd1ca77b7b87a8c",
+			"0xac5a288b16ed708b00cbe956a14e2a64b321f40851ea25ab1aded7b096822ec6",
+			"0xee95229b736827a21e0b7154873da3da59e38a1ba3e3b1770de2ed973495ce23",
+			"0x5168b6e3b5274054fec17b94a82931bc229d0270e5e4c9aeb61539fe8ba4f4a3",
+			"0x588c6144dfbb33922873946a44f03092bd65d2f89a01b3973002ed38f48917bc",
+			"0x6c51a964953dc036043b6e8c7fba4a3f6e83295306767bcc3562d9a8cdb04497",
+			"0x2cfdde11cc2669c5b504c4a82c31ff03e3f0b3aff044519b44bc39405bbf3c9b",
+			"0xd93200e550400a7179ac7afc394b22b7c710192eadbf429389450e7167191fdb",
+			"0x5e32f497c606c01c9be8ce1cca5f3cf475f62cf38b2195244a7c93df7d064dde",
+			"0xb0b74e34fd3983aef31a0584b6a7483ee936622b5fffbdce1411a9db6171c685",
+			"0x69d0a96a2eca8d6a5c32e700d3f910330930a3348e98e5b276013fcf70c0fe5f",
+			"0xf49cbb26f2463d38fde553e483de2d4bc6dd85efe2d0e56a2a79810df6af6db9",
+			"0xe58aa167a26eadf11257f1fcb5da7eef50f9985b590320cc6c015176965ea58e",
+			"0x582f08d14023a661a9d9dfd1db8324be88fc631cb9136e92f379ad7703584414",
+			"0xb9c5e584f5e231c2c41b41799a945d60b7ec22487e587ef55e2d9710489c3c00",
+			"0xd8b70c6657a5bf1fa4b67c4fc046c1eda284d677c610dbacaf0d2d84e4d782dd",
+			"0x1e29fb536e468abf66b59dd6f48d177a5100a192f1807afc12bb5c97368b8c95",
+			"0x595ac042acdb51370bf6bfd2b2058dda4781dd7ec330474defb2d5dd00e7f50e",
+			"0x3d00887c6509b148c5fadde21014aae94924fef363fd5723d80d6a2028df1de5",
+			"0x9632d419c9fdc7e676b130806cb9c86631aade0adc6a129d3769908ff97e7ef3",
+			"0x2f91c7ce1158228c8264a6147c6ca396b0d536cdde997a922cde2772b786743f",
+			"0xf4c731ac2e2612b7e7afcf3ba5911a58a03ef075133923218f54a736b25acb81",
+			"0x6748a5143ad25f0a461257a47eeab8f057782a438274ced580ca2bd3725d6be3",
+			"0x05a113b04baf81b6396dd32d59a53908d07a374ebdf17314ec334b3330a16697",
+			"0xb1bafaef874da14b8a9f883f89a896ec37d9af1b0958f5a3867780278bfe940d",
+			"0xd93c67d29befe531330c37dfa8b64db7339c8ac35b8cb674d741a414d90d77ef",
+			"0xb4319281a05f00968367a349e626ec63227412193c30aaface14ac6f7a480024",
+			"0xf6dce891724a49024535047bee44f889d5228066a931a8f40c18c7efed60f31b",
+			"0xccdeb308e3402c7dc72588871c23fb4f6fdc60e2a63230f628c5a2951878535b",
+			"0x90958033b917eea42ffb8bf077ccab8b19c2448113e8a128815ab3441846404b",
+			"0xed3436553039948094f33cffaff91a7e2b1eab86b638de5c8eb09538c8df0a8e",
+			"0x0910ba3093e37abaf70ffc0ef6a3b9b82e8c5f7ae497eec14b753f23d0a25343",
+			"0xdc78d9fceec3add33b92e1a0ee402b80cd87946e0def46f284d26eac46be6bcd",
+			"0x1d71a82b8d4b89f73747d8073a36acf8496757d35bb452e9cfd74d758774f492",
+			"0x5399612937722b6fbcb90c1768a92bb35acd4da6168814c1d9deb6156cf35134",
+			"0xc7156ee3591ab91824b2d4868d28baf11855a63a801b3793f99fdde7867eac7f",
+			"0xd25b288f3333ea8e33b33f42edf7b64976281e4840599aa49b07da39f895073a",
+			"0xc6596a87c544e84cfaa2baaa4fb0a93f3fa9d1e6938426ea05a418568a1589b0",
+			"0xdbf30aa7f1cb3a6d7661934d7ea2ec0665843ea1ec9b05fddd77fdb33fa7d834",
+			"0xfac2628cb5d70c177e8defbf62b0448297ecb13962638b9a26c5d05203cb2937",
+			"0xe8c5a65edee74d59e7b9e2fb4779485958d95303fff95bfebc0c1f4bf24ef9bf",
+			"0x67d21b95f40ca7c991f37981393dcec3218e70d5402d82ecbfcb21ae532872bb",
+			"0xba19c10601ed47bffb755f791325c50af9229d5c25294f315116fdb3640ca35a",
+			"0xf4c026f65e31d4606541c4e679d30d7336cfeeb81e9893465eb08c310792c061",
+			"0x5425ff602e06947bda22cfb687ace85c10896b5d48de144149cba26bac9a35ae",
+			"0xe11f82e350c209213da7d977e2de5199559ad3793a24a226cb4be367c0aa3c4a",
+			"0x13fb0194f473a4af296184d3293005dffcd28f1cf4b986bdaacede4a0a096fa6",
+			"0x3dab80c9fc23caf409e38791f07e107da3b5d0d9d4a64f87164e6d455660591b",
+			"0x5901803b234ef0445acfcd884baf0777970d3b600b80ab137dc1015a5b790885",
+			"0x3735e2d0ac017cd2c0570830615ed9eb642183fee5b2c2155e5fefc1d4de2561",
+			"0xea9030afc249a1e4f6c86f4b19660ea63783824e8d1c3de30e38802f1aea9d12",
+			"0x9def3e76927de026a1a76f39c242263ffbf94b3ef42fb6e98626b67ec3d8a308",
+			"0x504886149f29d5eb02412202f04b45f23cc97a82206bdf237c433e83c52e386b",
+			"0xc84d3b9e84727c2d9279dfebcdf8104762e3b101dd9f39c9482a456f87bcf976",
+			"0xf4737cd864b198c8049ece5a4f18eea18b48a649997c42504b560188adfb6f2b",
+			"0x410894a208ff99bd3ec5f925aa7ff6932c4404390df9e337601fa7b7251d09ff",
+			"0xe6e7e7e2b54bcbe37eb7066b9a3c9b01b632d228c8ed5ef8f1871d9b7235a55b",
+			"0x8af02fe4657f956a9ede1a9b0a83d67367b43a8f5efdf6bd753d09f428b58960",
+			"0xdd056794fcb6f5b694934b17ef35577dbec8d04159fe447e0eba02e4ff4e9d96",
+			"0xef94a3289e9e197b04c17886373adbe66a451efe052fb419846006b5f659b91a",
+			"0x5558e5612dc3c222d76e7190409fc5d38479e08f6fdf4b7df299e6da968c2811",
+			"0x5f38cc5046a503d424c4b63cb060956babe0c1de050517d0067eccfa23cb664d",
+			"0x4c6a04827f46e51378e9ff3b811c878dead62da3ae20ef7d392c57512a09a719",
+			"0x68fd9806bea1d937063778b245a6788cd992e4fc614d5bce1ffe06e125f67701",
+			"0x0751a79f86c4220e81bbf6b4692976ed4c315f9b62937b23083f4db2370f04eb",
+			"0x5651840a2065a3468d97bf824e78455eb5e22b82b45fde7bb69b297c4c40e853",
+			"0x959d277feeb492ed3393fcd5774f1dde27c6018b07cc4fc93b937ee266ae12ea",
+			"0xa145f2ed353350a89de97e83433188adbd14107c16c654c131203442c34f2899",
+			"0xf888fc918eab6f88a5f6c6545d47a32dd558f10d00e86a9949cb3f144b7264e2",
+			"0x2fef7f5f1c149c87602cbe38766631abc992efaee26dd5b59e5bc361f6901cf4",
+			"0x5e0735a9be3eb97e790a93502a6f138a8863b0f0c20905bee0ac14d5aff4206a",
+			"0x9c40cbdab8e769c2c6f9de57c23621c5d8a25adaedd1ba9df17fa1680cd4e63f",
+			"0x995e6a00f5bc63f0a5d837a6671260771f12995b82da19c72a9546ae7b93f38f",
+			"0x10f647c754615b475543fca5ed9a4491503867fae619513dad966af63cb7ed2b",
+			"0x7e45b18bd06e946d20bc3807aa926aa5c0a10555b9dc18f9574136969b8f48d5",
+			"0x8a419fcb515eef5a139a37241872c095818fd276f77e6388c74b82ae49e6386f",
+			"0x2cb2d82e278d8ba47830ed3dbd6edf69d2e49721182488ffe0a01698071e7d40",
+			"0x1e9e5cb7a68e7a2decef900bd250bd31c1f024ddd240799327263c72940c7e59",
+			"0x3904821c4388b16e9851e3a7318f4177102da38525cd3e5e5708430a243470a1",
+			"0x7785b9d9ff6dfa45d945aaa393270bf8b62bbde46e941aa068606c173963e518",
+			"0xbff782fbbadfc752f5c2a239e49ab13200fc35cfba29a5332fe936a0367d336c",
+			"0x294a3fd73f102c7017c9208857dfb91ec7c323916dd1bb3c9f1fac2c7d952b4b",
+			"0x3a9fe3685814b6cff7a918e514a9d5375a1fc8268a48d5b78da93dfda115332f",
+			"0xc033a1836983d485e28e5a0825c953a76d4c7f4a8ec5f7eb0b11baf5ae2beb79",
+			"0xb088df91127cad5d72f8db96d7ea86dfdef55374bf982222a45df7d2b631ceb6"
 		]
 	},
 	"nodes": [

ethcore/res/ethereum/tests

@@ -1 +1 @@
-Subproject commit 3f5febc901913ef698f1b09dda8705babd729e4a
+Subproject commit 2cd62aeec11da29766b30d500f2b9a96f1f28cf0

ethcore/res/ethereum/tests-issues/currents.json

@@ -1,468 +1,42 @@
 { "block":
 	[
 		{
-			"reference": "9590",
-			"failing": "stCreateTest",
-			"subtests": [
-				"CreateOOGafterInitCodeReturndata2_d0g1v0_Constantinople"
-			]
+			"reference": "None",
+			"comment": "This failing test is deemed skippable. Could not happen on a mainnet.",
+			"failing": "GeneralStateTest_stCreate2",
+			"subtests": ["RevertInCreateInInitCreate2_d0g0v0_Constantinople"]
 		},
 		{
-			"reference": "9590",
-			"failing": "stCreate2",
-			"subtests": [
-				"RevertDepthCreateAddressCollision_d0g1v0_Constantinople",
-				"RevertDepthCreateAddressCollision_d1g1v1_Constantinople",
-				"CREATE2_Suicide_d5g0v0_Constantinople",
-				"CREATE2_Suicide_d7g0v0_Constantinople",
-				"create2collisionSelfdestructedOOG_d2g0v0_Byzantium",
-				"create2collisionSelfdestructedOOG_d2g0v0_Constantinople",
-				"create2collisionNonce_d1g0v0_Byzantium",
-				"create2collisionNonce_d1g0v0_Constantinople",
-				"CreateMessageRevertedOOGInInit_d0g1v0_Constantinople",
-				"create2callPrecompiles_d3g0v0_Constantinople",
-				"create2collisionCode_d1g0v0_Byzantium",
-				"create2collisionCode_d1g0v0_Constantinople",
-				"create2collisionStorage_d0g0v0_Byzantium",
-				"create2collisionStorage_d0g0v0_Constantinople",
-				"create2callPrecompiles_d4g0v0_Constantinople",
-				"create2collisionSelfdestructedRevert_d0g0v0_Byzantium",
-				"create2collisionSelfdestructedRevert_d0g0v0_Constantinople",
-				"CreateMessageReverted_d0g1v0_Constantinople",
-				"RevertOpcodeCreate_d0g1v0_Constantinople",
-				"CREATE2_Suicide_d11g0v0_Constantinople",
-				"create2checkFieldsInInitcode_d5g0v0_Constantinople",
-				"create2collisionSelfdestructedOOG_d1g0v0_Byzantium",
-				"create2collisionSelfdestructedOOG_d1g0v0_Constantinople",
-				"returndatacopy_following_create_d1g0v0_Constantinople",
-				"RevertDepthCreate2OOG_d1g1v1_Constantinople",
-				"create2collisionSelfdestructed_d2g0v0_Byzantium",
-				"create2collisionSelfdestructed_d2g0v0_Constantinople",
-				"create2callPrecompiles_d2g0v0_Constantinople",
-				"create2InitCodes_d2g0v0_Constantinople",
-				"create2collisionNonce_d2g0v0_Byzantium",
-				"create2collisionNonce_d2g0v0_Constantinople",
-				"create2collisionCode_d0g0v0_Byzantium",
-				"create2collisionCode_d0g0v0_Constantinople",
-				"CREATE2_Bounds_d0g0v0_Constantinople",
-				"RevertDepthCreate2OOG_d0g0v0_Constantinople",
-				"CREATE2_Suicide_d1g0v0_Constantinople",
-				"CREATE2_Bounds3_d0g1v0_Constantinople",
-				"create2collisionStorage_d2g0v0_Byzantium",
-				"create2collisionStorage_d2g0v0_Constantinople",
-				"RevertDepthCreateAddressCollision_d0g0v1_Constantinople",
-				"create2callPrecompiles_d5g0v0_Constantinople",
-				"create2collisionCode2_d0g0v0_Byzantium",
-				"create2collisionCode2_d0g0v0_Constantinople",
-				"create2noCash_d0g0v0_Byzantium",
-				"create2noCash_d0g0v0_Constantinople",
-				"create2checkFieldsInInitcode_d7g0v0_Constantinople",
-				"create2SmartInitCode_d1g0v0_Constantinople",
-				"create2InitCodes_d6g0v0_Constantinople",
-				"create2noCash_d1g0v0_Byzantium",
-				"create2noCash_d1g0v0_Constantinople",
-				"CREATE2_ContractSuicideDuringInit_ThenStoreThenReturn_d0g0v0_Constantinople",
-				"RevertOpcodeInCreateReturns_d0g0v0_Constantinople",
-				"create2collisionStorage_d1g0v0_Byzantium",
-				"create2collisionStorage_d1g0v0_Constantinople",
-				"create2checkFieldsInInitcode_d3g0v0_Constantinople",
-				"create2collisionBalance_d0g0v0_Byzantium",
-				"create2collisionBalance_d0g0v0_Constantinople",
-				"create2collisionSelfdestructed2_d0g0v0_Constantinople",
-				"create2InitCodes_d3g0v0_Constantinople",
-				"create2collisionCode2_d1g0v0_Byzantium",
-				"create2collisionCode2_d1g0v0_Constantinople",
-				"create2checkFieldsInInitcode_d1g0v0_Constantinople",
-				"create2collisionBalance_d1g0v0_Byzantium",
-				"create2collisionBalance_d1g0v0_Constantinople",
-				"CREATE2_Bounds3_d0g2v0_Constantinople",
-				"create2callPrecompiles_d6g0v0_Constantinople",
-				"Create2Recursive_d0g0v0_Constantinople",
-				"create2collisionSelfdestructedOOG_d0g0v0_Byzantium",
-				"create2collisionSelfdestructedOOG_d0g0v0_Constantinople",
-				"CREATE2_Suicide_d3g0v0_Constantinople",
-				"returndatacopy_following_create_d0g0v0_Constantinople",
-				"create2InitCodes_d8g0v0_Constantinople",
-				"RevertDepthCreate2OOG_d0g0v1_Constantinople",
-				"create2checkFieldsInInitcode_d2g0v0_Constantinople",
-				"RevertDepthCreate2OOG_d1g0v1_Constantinople",
-				"Create2OnDepth1024_d0g0v0_Constantinople",
-				"create2collisionSelfdestructed2_d1g0v0_Constantinople",
-				"create2collisionSelfdestructedRevert_d2g0v0_Byzantium",
-				"create2collisionSelfdestructedRevert_d2g0v0_Constantinople",
-				"create2callPrecompiles_d0g0v0_Constantinople",
-				"RevertDepthCreateAddressCollision_d0g1v1_Constantinople",
-				"create2collisionSelfdestructed_d1g0v0_Byzantium",
-				"create2collisionSelfdestructed_d1g0v0_Constantinople",
-				"call_outsize_then_create2_successful_then_returndatasize_d0g0v0_Byzantium",
-				"call_outsize_then_create2_successful_then_returndatasize_d0g0v0_Constantinople",
-				"Create2OOGafterInitCodeRevert_d0g0v0_Constantinople",
-				"Create2OOGafterInitCodeReturndata3_d0g0v0_Constantinople",
-				"Create2OOGafterInitCodeReturndataSize_d0g0v0_Constantinople",
-				"create2InitCodes_d7g0v0_Constantinople",
-				"CREATE2_Suicide_d10g0v0_Constantinople",
-				"RevertDepthCreate2OOG_d0g1v0_Constantinople",
-				"create2InitCodes_d5g0v0_Constantinople",
-				"create2collisionSelfdestructedRevert_d1g0v0_Byzantium",
-				"create2collisionSelfdestructedRevert_d1g0v0_Constantinople",
-				"RevertDepthCreate2OOG_d1g1v0_Constantinople",
-				"create2collisionSelfdestructed_d0g0v0_Byzantium",
-				"create2collisionSelfdestructed_d0g0v0_Constantinople",
-				"create2noCash_d2g0v0_Byzantium",
-				"create2noCash_d2g0v0_Constantinople",
-				"CREATE2_Bounds3_d0g0v0_Constantinople",
-				"create2collisionNonce_d0g0v0_Byzantium",
-				"create2collisionNonce_d0g0v0_Constantinople",
-				"CREATE2_Suicide_d2g0v0_Constantinople",
-				"Create2OOGafterInitCode_d0g0v0_Constantinople",
-				"call_then_create2_successful_then_returndatasize_d0g0v0_Byzantium",
-				"call_then_create2_successful_then_returndatasize_d0g0v0_Constantinople",
-				"create2collisionBalance_d2g0v0_Byzantium",
-				"create2collisionBalance_d2g0v0_Constantinople",
-				"create2checkFieldsInInitcode_d6g0v0_Constantinople",
-				"RevertDepthCreate2OOG_d0g1v1_Constantinople",
-				"returndatacopy_afterFailing_create_d0g0v0_Constantinople",
-				"returndatacopy_following_revert_in_create_d0g0v0_Constantinople",
-				"CREATE2_Suicide_d9g0v0_Constantinople",
-				"create2callPrecompiles_d7g0v0_Constantinople",
-				"RevertDepthCreateAddressCollision_d1g0v1_Constantinople",
-				"create2InitCodes_d1g0v0_Constantinople",
-				"CREATE2_Bounds_d0g1v0_Constantinople",
-				"Create2OOGafterInitCodeReturndata_d0g0v0_Constantinople",
-				"create2checkFieldsInInitcode_d4g0v0_Constantinople",
-				"CreateMessageRevertedOOGInInit_d0g0v0_Constantinople",
-				"RevertDepthCreateAddressCollision_d1g1v0_Constantinople",
-				"returndatacopy_following_successful_create_d0g0v0_Constantinople",
-				"create2checkFieldsInInitcode_d0g0v0_Constantinople",
-				"CreateMessageReverted_d0g0v0_Constantinople",
-				"create2SmartInitCode_d0g0v0_Constantinople",
-				"CREATE2_Bounds2_d0g0v0_Constantinople",
-				"returndatasize_following_successful_create_d0g0v0_Constantinople",
-				"CREATE2_Bounds2_d0g1v0_Constantinople",
-				"returndatacopy_0_0_following_successful_create_d0g0v0_Constantinople",
-				"RevertDepthCreateAddressCollision_d0g0v0_Constantinople",
-				"CREATE2_Suicide_d0g0v0_Constantinople",
-				"create2InitCodes_d0g0v0_Constantinople",
-				"Create2OnDepth1023_d0g0v0_Constantinople",
-				"create2InitCodes_d4g0v0_Constantinople",
-				"Create2OOGafterInitCodeReturndata2_d0g0v0_Constantinople",
-				"create2collisionBalance_d3g0v0_Byzantium",
-				"create2collisionBalance_d3g0v0_Constantinople",
-				"CREATE2_Suicide_d4g0v0_Constantinople",
-				"Create2OOGafterInitCode_d0g1v0_Constantinople",
-				"RevertDepthCreateAddressCollision_d1g0v0_Constantinople",
-				"Create2OOGafterInitCodeRevert2_d0g0v0_Constantinople",
-				"Create2OOGafterInitCodeReturndata_d0g1v0_Constantinople",
-				"Create2Recursive_d0g1v0_Constantinople",
-				"create2collisionCode_d2g0v0_Byzantium",
-				"create2collisionCode_d2g0v0_Constantinople",
-				"CREATE2_Suicide_d6g0v0_Constantinople",
-				"CREATE2_Suicide_d8g0v0_Constantinople",
-				"RevertOpcodeCreate_d0g0v0_Constantinople",
-				"Create2OOGafterInitCodeReturndata2_d0g1v0_Constantinople",
-				"create2callPrecompiles_d1g0v0_Constantinople",
-				"RevertInCreateInInit_d0g0v0_Constantinople",
-				"RevertDepthCreate2OOG_d1g0v0_Constantinople"
-			]
-		},
-		{
-			"reference": "9590",
-			"failing": "bcStateTest",
-			"subtests": [
-				"suicideStorageCheck_Byzantium",
-				"suicideStorageCheck_Constantinople",
-				"suicideStorageCheckVCreate2_Byzantium",
-				"suicideStorageCheckVCreate2_Constantinople",
-				"create2collisionwithSelfdestructSameBlock_Constantinople",
-				"blockhashNonConstArg_Constantinople",
-				"suicideThenCheckBalance_Constantinople",
-				"suicideThenCheckBalance_Homestead",
-				"suicideStorageCheckVCreate_Byzantium",
-				"suicideStorageCheckVCreate_Constantinople"
-			]
-		},
-		{
-			"reference": "9590",
-			"failing": "stEIP158Specific",
-			"subtests": [
-				"callToEmptyThenCallError_d0g0v0_Byzantium",
-				"callToEmptyThenCallError_d0g0v0_Constantinople",
-				"callToEmptyThenCallError_d0g0v0_EIP158"
-			]
-		},
-		{
-			"reference": "9590",
-			"failing": "stPreCompiledContracts",
-			"subtests": [
-				"identity_to_smaller_d0g0v0_Constantinople",
-				"identity_to_bigger_d0g0v0_Constantinople"
-			]
-		},
-		{
-			"reference": "9590",
-			"failing": "stReturnDataTest",
-			"subtests": [
-				"modexp_modsize0_returndatasize_d0g1v0_Constantinople",
-				"modexp_modsize0_returndatasize_d0g2v0_Constantinople",
-				"modexp_modsize0_returndatasize_d0g3v0_Constantinople"
-			]
-		},
-		{
-			"reference": "9590",
-			"failing": "stSpecialTest",
-			"subtests": [
-				"push32withoutByte_d0g0v0_Constantinople"
-			]
+			"reference": "None",
+			"comment": "This failing test is deemed skippable. Could not happen on a mainnet.",
+			"failing": "GeneralStateTest_stRevertTest",
+			"subtests": ["RevertInCreateInInit_d0g0v0_Constantinople"]
 		}
-
 	],
 	"state":
 	[
 		{
-			"reference": "9590",
-			"failing": "stCreateTest",
+			"reference": "None",
+			"comment": "This failing test is deemed skippable. Could not happen on a mainnet.",
+			"failing": "stCreate2Test",
 			"subtests": {
-				"CreateOOGafterInitCodeReturndata2": {
-					"subnumbers": ["2"],
+				"RevertInCreateInInitCreate2": {
+					"subnumbers": ["1"],
 					"chain": "Constantinople (test)"
 				}
 			}
 		},
 		{
-			"reference": "9590",
-			"failing": "stCreate2Test",
+			"reference": "None",
+			"comment": "This failing test is deemed skippable. Could not happen on a mainnet.",
+			"failing": "stRevertTest",
 			"subtests": {
 				"RevertInCreateInInit": {
 					"subnumbers": ["1"],
 					"chain": "Constantinople (test)"
 				}
 			}
-		},
-		{
-			"reference": "9590",
-			"failing": "stEIP150Specific",
-			"subtests": {
-				"NewGasPriceForCodes": {
-					"subnumbers": ["1"],
-					"chain": "Constantinople (test)"
-				}
-			}
-		},
-		{
-			"reference": "9590",
-			"failing": "stInitCodeTest",
-			"subtests": {
-				"OutOfGasContractCreation": {
-					"subnumbers": ["4"],
-					"chain": "Constantinople (test)"
-				}
-			}
-		},
-		{
-			"reference": "9590",
-			"failing": "stPreCompiledContracts",
-			"subtests": {
-				"modexp": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				}
-			}
-		},
-		{
-			"reference": "9590",
-			"failing": "stRevertTest",
-			"subtests": {
-				"LoopCallsDepthThenRevert3": {
-					"subnumbers": ["1"],
-					"chain": "Constantinople (test)"
-				},
-				"RevertOpcodeCreate": {
-					"subnumbers": ["1"],
-					"chain": "Constantinople (test)"
-				},
-				"RevertSubCallStorageOOG2": {
-					"subnumbers": ["1","3"],
-					"chain": "Constantinople (test)"
-				},
-				"RevertDepthCreateOOG": {
-					"subnumbers": ["3","4"],
-					"chain": "Constantinople (test)"
-				},
-				"RevertOpcodeMultipleSubCalls": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"RevertOpcodeDirectCall": {
-					"subnumbers": ["1","2"],
-					"chain": "Constantinople (test)"
-				},
-				"LoopCallsDepthThenRevert2": {
-					"subnumbers": ["1"],
-					"chain": "Constantinople (test)"
-				},
-				"RevertDepth2": {
-					"subnumbers": ["1"],
-					"chain": "Constantinople (test)"
-				},
-				"RevertRemoteSubCallStorageOOG2": {
-					"subnumbers": ["1","2"],
-					"chain": "Constantinople (test)"
-				},
-				"RevertDepthCreateAddressCollision": {
-					"subnumbers": ["3","4"],
-					"chain": "Constantinople (test)"
-				}
-			}
-		},
-		{
-			"reference": "9590",
-			"failing": "stStaticCall",
-			"subtests": {
-				"static_RevertDepth2": {
-					"subnumbers": ["1","3"],
-					"chain": "Constantinople (test)"
-				},
-				"static_CheckOpcodes4": {
-					"subnumbers": ["3"],
-					"chain": "Constantinople (test)"
-				},
-				"static_CheckOpcodes3": {
-					"subnumbers": ["5","6","7","8"],
-					"chain": "Constantinople (test)"
-				},
-				"static_callBasic": {
-					"subnumbers": ["1","2"],
-					"chain": "Constantinople (test)"
-				},
-				"static_CheckOpcodes2": {
-					"subnumbers": ["5","6","7","8"],
-					"chain": "Constantinople (test)"
-				},
-				"static_callCreate": {
-					"subnumbers": ["2"],
-					"chain": "Constantinople (test)"
-				}
-			}
-		},
-		{
-			"reference": "https://github.com/ethereum/tests/issues/512",
-			"failing": "stZeroKnowledge",
-			"subtests": {
-				"pointAddTrunc": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"pointAdd": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"pointMulAdd": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"pointMulAdd2": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				}
-			}
-		},
-		{
-			"reference": "9590",
-			"failing": "stCreate2Test",
-			"subtests": {
-				"call_then_create2_successful_then_returndatasize": {
-					"subnumbers": ["1"],
-					"chain": "Constantinople (test)"
-				},
-				"returndatacopy_afterFailing_create": {
-					"subnumbers": ["1"],
-					"chain": "Constantinople (test)"
-				},
-				"create2checkFieldsInInitcode": {
-					"subnumbers": ["1","2","3","5","6","7"],
-					"chain": "Constantinople (test)"
-				},
-				"Create2Recursive": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"create2collisionBalance": {
-					"subnumbers": ["2","3"],
-					"chain": "Constantinople (test)"
-				},
-				"create2InitCodes": {
-					"subnumbers": ["1","5","6","7","8","9"],
-					"chain": "Constantinople (test)"
-				},
-				"Create2OOGafterInitCode": {
-					"subnumbers": ["2"],
-					"chain": "Constantinople (test)"
-				},
-				"CreateMessageRevertedOOGInInit": {
-					"subnumbers": ["2"],
-					"chain": "Constantinople (test)"
-				},
-				"returndatacopy_following_revert_in_create": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"create2collisionSelfdestructed": {
-					"subnumbers": ["2"],
-					"chain": "Constantinople (test)"
-				},
-				"returndatacopy_0_0_following_successful_create": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"Create2OnDepth1023": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"Create2OOGafterInitCodeReturndata2": {
-					"subnumbers": ["2"],
-					"chain": "Constantinople (test)"
-				},
-				"RevertOpcodeInCreateReturns": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"CREATE2_ContractSuicideDuringInit_ThenStoreThenReturn": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"returndatasize_following_successful_create": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"call_outsize_then_create2_successful_then_returndatasize": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"CreateMessageReverted": {
-					"subnumbers": ["2"],
-					"chain": "Constantinople (test)"
-				},
-				"CREATE2_Suicide": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"Create2OOGafterInitCodeRevert": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"Create2OnDepth1024": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				},
-				"create2collisionStorage": {
-					"subnumbers": ["2","3"],
-					"chain": "Constantinople (test)"
-				},
-				"create2callPrecompiles": {
-					"subnumbers": ["*"],
-					"chain": "Constantinople (test)"
-				}
-			}
 		}
+		
 	]
 }

ethcore/service/src/service.rs

@@ -106,7 +106,13 @@ impl ClientService {
 		info!("Configured for {} using {} engine", Colour::White.bold().paint(spec.name.clone()), Colour::Yellow.bold().paint(spec.engine.name()));
 
 		let pruning = config.pruning;
-		let client = Client::new(config, &spec, blockchain_db.clone(), miner.clone(), io_service.channel())?;
+		let client = Client::new(
+			config,
+			&spec,
+			blockchain_db.clone(),
+			miner.clone(),
+			io_service.channel(),
+		)?;
 		miner.set_io_channel(io_service.channel());
 		miner.set_in_chain_checker(&client.clone());
 
@@ -117,7 +123,7 @@ impl ClientService {
 			pruning: pruning,
 			channel: io_service.channel(),
 			snapshot_root: snapshot_path.into(),
-			db_restore: client.clone(),
+			client: client.clone(),
 		};
 		let snapshot = Arc::new(SnapshotService::new(snapshot_params)?);
 

ethcore/src/blockchain/blockchain.rs

@@ -229,6 +229,7 @@ pub struct BlockChain {
 
 	cache_man: Mutex<CacheManager<CacheId>>,
 
+	pending_best_ancient_block: RwLock<Option<Option<BestAncientBlock>>>,
 	pending_best_block: RwLock<Option<BestBlock>>,
 	pending_block_hashes: RwLock<HashMap<BlockNumber, H256>>,
 	pending_block_details: RwLock<HashMap<H256, BlockDetails>>,
@@ -538,6 +539,7 @@ impl BlockChain {
 			block_receipts: RwLock::new(HashMap::new()),
 			db: db.clone(),
 			cache_man: Mutex::new(cache_man),
+			pending_best_ancient_block: RwLock::new(None),
 			pending_best_block: RwLock::new(None),
 			pending_block_hashes: RwLock::new(HashMap::new()),
 			pending_block_details: RwLock::new(HashMap::new()),
@@ -808,18 +810,7 @@ impl BlockChain {
 			}, is_best);
 
 			if is_ancient {
-				let mut best_ancient_block = self.best_ancient_block.write();
-				let ancient_number = best_ancient_block.as_ref().map_or(0, |b| b.number);
-				if self.block_hash(block_number + 1).is_some() {
-					batch.delete(db::COL_EXTRA, b"ancient");
-					*best_ancient_block = None;
-				} else if block_number > ancient_number {
-					batch.put(db::COL_EXTRA, b"ancient", &hash);
-					*best_ancient_block = Some(BestAncientBlock {
-						hash: hash,
-						number: block_number,
-					});
-				}
+				self.set_best_ancient_block(block_number, &hash, batch);
 			}
 
 			false
@@ -860,6 +851,84 @@ impl BlockChain {
 		}
 	}
 
+	/// Update the best ancient block to the given hash, after checking that
+	/// it's directly linked to the currently known best ancient block
+	pub fn update_best_ancient_block(&self, hash: &H256) {
+		// Get the block view of the next ancient block (it must
+		// be in DB at this point)
+		let block_view = match self.block(hash) {
+			Some(v) => v,
+			None => return,
+		};
+
+		// So that `best_ancient_block` gets unlocked before calling
+		// `set_best_ancient_block`
+		{
+			// Get the target hash ; if there are no ancient block,
+			// it means that the chain is already fully linked
+			// Release the `best_ancient_block` RwLock
+			let target_hash = {
+				let best_ancient_block = self.best_ancient_block.read();
+				let cur_ancient_block = match *best_ancient_block {
+					Some(ref b) => b,
+					None => return,
+				};
+
+				// Ensure that the new best ancient block is after the current one
+				if block_view.number() <= cur_ancient_block.number {
+					return;
+				}
+
+				cur_ancient_block.hash.clone()
+			};
+
+			let mut block_hash = *hash;
+			let mut is_linked = false;
+
+			loop {
+				if block_hash == target_hash {
+					is_linked = true;
+					break;
+				}
+
+				match self.block_details(&block_hash) {
+					Some(block_details) => {
+						block_hash = block_details.parent;
+					},
+					None => break,
+				}
+			}
+
+			if !is_linked {
+				trace!(target: "blockchain", "The given block {:x} is not linked to the known ancient block {:x}", hash, target_hash);
+				return;
+			}
+		}
+
+		let mut batch = self.db.key_value().transaction();
+		self.set_best_ancient_block(block_view.number(), hash, &mut batch);
+		self.db.key_value().write(batch).expect("Low level database error.");
+	}
+
+	/// Set the best ancient block with the given value: private method
+	/// `best_ancient_block` must not be locked, otherwise a DeadLock would occur
+	fn set_best_ancient_block(&self, block_number: BlockNumber, block_hash: &H256, batch: &mut DBTransaction) {
+		let mut pending_best_ancient_block = self.pending_best_ancient_block.write();
+		let ancient_number = self.best_ancient_block.read().as_ref().map_or(0, |b| b.number);
+		if self.block_hash(block_number + 1).is_some() {
+			trace!(target: "blockchain", "The two ends of the chain have met.");
+			batch.delete(db::COL_EXTRA, b"ancient");
+			*pending_best_ancient_block = Some(None);
+		} else if block_number > ancient_number {
+			trace!(target: "blockchain", "Updating the best ancient block to {}.", block_number);
+			batch.put(db::COL_EXTRA, b"ancient", &block_hash);
+			*pending_best_ancient_block = Some(Some(BestAncientBlock {
+				hash: *block_hash,
+				number: block_number,
+			}));
+		}
+	}
+
 	/// Insert an epoch transition. Provide an epoch number being transitioned to
 	/// and epoch transition object.
 	///
@@ -1112,15 +1181,21 @@ impl BlockChain {
 
 	/// Apply pending insertion updates
 	pub fn commit(&self) {
+		let mut pending_best_ancient_block = self.pending_best_ancient_block.write();
 		let mut pending_best_block = self.pending_best_block.write();
 		let mut pending_write_hashes = self.pending_block_hashes.write();
 		let mut pending_block_details = self.pending_block_details.write();
 		let mut pending_write_txs = self.pending_transaction_addresses.write();
 
 		let mut best_block = self.best_block.write();
+		let mut best_ancient_block = self.best_ancient_block.write();
 		let mut write_block_details = self.block_details.write();
 		let mut write_hashes = self.block_hashes.write();
 		let mut write_txs = self.transaction_addresses.write();
+		// update best ancient block
+		if let Some(block_option) = pending_best_ancient_block.take() {
+			*best_ancient_block = block_option;
+		}
 		// update best block
 		if let Some(block) = pending_best_block.take() {
 			*best_block = block;

ethcore/src/client/chain_notify.rs

@@ -15,7 +15,7 @@
 // along with Parity.  If not, see <http://www.gnu.org/licenses/>.
 
 use bytes::Bytes;
-use ethereum_types::H256;
+use ethereum_types::{H256, U256};
 use transaction::UnverifiedTransaction;
 use blockchain::ImportRoute;
 use std::time::Duration;
@@ -141,7 +141,15 @@ pub trait ChainNotify : Send + Sync {
 	}
 
 	/// fires when chain broadcasts a message
-	fn broadcast(&self, _message_type: ChainMessageType) {}
+	fn broadcast(&self, _message_type: ChainMessageType) {
+		// does nothing by default
+	}
+
+	/// fires when new block is about to be imported
+	/// implementations should be light
+	fn block_pre_import(&self, _bytes: &Bytes, _hash: &H256, _difficulty: &U256) {
+		// does nothing by default
+	}
 
 	/// fires when new transactions are received from a peer
 	fn transactions_received(&self,

ethcore/src/client/client.rs

@@ -32,7 +32,7 @@ use kvdb::{DBValue, KeyValueDB, DBTransaction};
 // other
 use ethereum_types::{H256, Address, U256};
 use block::{IsBlock, LockedBlock, Drain, ClosedBlock, OpenBlock, enact_verified, SealedBlock};
-use blockchain::{BlockChain, BlockChainDB, BlockProvider, TreeRoute, ImportRoute, TransactionAddress, ExtrasInsert};
+use blockchain::{BlockReceipts, BlockChain, BlockChainDB, BlockProvider, TreeRoute, ImportRoute, TransactionAddress, ExtrasInsert};
 use client::ancient_import::AncientVerifier;
 use client::{
 	Nonce, Balance, ChainInfo, BlockInfo, CallContract, TransactionInfo,
@@ -66,7 +66,7 @@ use ethcore_miner::pool::VerifiedTransaction;
 use parking_lot::{Mutex, RwLock};
 use rand::OsRng;
 use receipt::{Receipt, LocalizedReceipt};
-use snapshot::{self, io as snapshot_io};
+use snapshot::{self, io as snapshot_io, SnapshotClient};
 use spec::Spec;
 use state_db::StateDB;
 use state::{self, State};
@@ -481,7 +481,7 @@ impl Importer {
 
 		let mut batch = DBTransaction::new();
 
-		let ancestry_actions = self.engine.ancestry_actions(&block, &mut chain.ancestry_with_metadata_iter(*parent));
+		let ancestry_actions = self.engine.ancestry_actions(&header, &mut chain.ancestry_with_metadata_iter(*parent));
 
 		let receipts = block.receipts;
 		let traces = block.traces.drain();
@@ -881,7 +881,7 @@ impl Client {
 	/// Flush the block import queue.
 	pub fn flush_queue(&self) {
 		self.importer.block_queue.flush();
-		while !self.importer.block_queue.queue_info().is_empty() {
+		while !self.importer.block_queue.is_empty() {
 			self.import_verified_blocks();
 		}
 	}
@@ -1005,6 +1005,16 @@ impl Client {
 		self.importer.miner.clone()
 	}
 
+	#[cfg(test)]
+	pub fn state_db(&self) -> ::parking_lot::RwLockReadGuard<StateDB> {
+		self.state_db.read()
+	}
+
+	#[cfg(test)]
+	pub fn chain(&self) -> Arc<BlockChain> {
+		self.chain.read().clone()
+	}
+
 	/// Replace io channel. Useful for testing.
 	pub fn set_io_channel(&self, io_channel: IoChannel<ClientIoMessage>) {
 		*self.io_channel.write() = io_channel;
@@ -1413,8 +1423,21 @@ impl ImportBlock for Client {
 			bail!(EthcoreErrorKind::Block(BlockError::UnknownParent(unverified.parent_hash())));
 		}
 
+		let raw = if self.importer.block_queue.is_empty() {
+			Some((
+				unverified.bytes.clone(),
+				unverified.header.hash(),
+				*unverified.header.difficulty(),
+			))
+		} else { None };
+
 		match self.importer.block_queue.import(unverified) {
-			Ok(res) => Ok(res),
+			Ok(hash) => {
+				if let Some((raw, hash, difficulty)) = raw {
+					self.notify(move |n| n.block_pre_import(&raw, &hash, &difficulty));
+				}
+				Ok(hash)
+			},
 			// we only care about block errors (not import errors)
 			Err((block, EthcoreError(EthcoreErrorKind::Block(err), _))) => {
 				self.importer.bad_blocks.report(block.bytes, format!("{:?}", err));
@@ -1817,7 +1840,7 @@ impl BlockChainClient for Client {
 		Some(receipt)
 	}
 
-	fn block_receipts(&self, id: BlockId) -> Option<Vec<LocalizedReceipt>> {
+	fn localized_block_receipts(&self, id: BlockId) -> Option<Vec<LocalizedReceipt>> {
 		let hash = self.block_hash(id)?;
 
 		let chain = self.chain.read();
@@ -1860,14 +1883,18 @@ impl BlockChainClient for Client {
 		self.state_db.read().journal_db().state(hash)
 	}
 
-	fn encoded_block_receipts(&self, hash: &H256) -> Option<Bytes> {
-		self.chain.read().block_receipts(hash).map(|receipts| ::rlp::encode(&receipts))
+	fn block_receipts(&self, hash: &H256) -> Option<BlockReceipts> {
+		self.chain.read().block_receipts(hash)
 	}
 
 	fn queue_info(&self) -> BlockQueueInfo {
 		self.importer.block_queue.queue_info()
 	}
 
+	fn is_queue_empty(&self) -> bool {
+		self.importer.block_queue.is_empty()
+	}
+
 	fn clear_queue(&self) {
 		self.importer.block_queue.clear();
 	}
@@ -2277,26 +2304,37 @@ impl ScheduleInfo for Client {
 
 impl ImportSealedBlock for Client {
 	fn import_sealed_block(&self, block: SealedBlock) -> EthcoreResult<H256> {
-		let h = block.header().hash();
 		let start = Instant::now();
+		let raw = block.rlp_bytes();
+		let header = block.header().clone();
+		let hash = header.hash();
+		self.notify(|n| n.block_pre_import(&raw, &hash, header.difficulty()));
+
 		let route = {
+			// Do a super duper basic verification to detect potential bugs
+			if let Err(e) = self.engine.verify_block_basic(&header) {
+				self.importer.bad_blocks.report(
+					block.rlp_bytes(),
+					format!("Detected an issue with locally sealed block: {}", e),
+				);
+				return Err(e.into());
+			}
+
 			// scope for self.import_lock
 			let _import_lock = self.importer.import_lock.lock();
 			trace_time!("import_sealed_block");
 
-			let number = block.header().number();
 			let block_data = block.rlp_bytes();
-			let header = block.header().clone();
 
 			let route = self.importer.commit_block(block, &header, encoded::Block::new(block_data), self);
-			trace!(target: "client", "Imported sealed block #{} ({})", number, h);
+			trace!(target: "client", "Imported sealed block #{} ({})", header.number(), hash);
 			self.state_db.write().sync_cache(&route.enacted, &route.retracted, false);
 			route
 		};
 		let route = ChainRoute::from([route].as_ref());
 		self.importer.miner.chain_new_blocks(
 			self,
-			&[h.clone()],
+			&[hash],
 			&[],
 			route.enacted(),
 			route.retracted(),
@@ -2304,16 +2342,16 @@ impl ImportSealedBlock for Client {
 		);
 		self.notify(|notify| {
 			notify.new_blocks(
-				vec![h.clone()],
+				vec![hash],
 				vec![],
 				route.clone(),
-				vec![h.clone()],
+				vec![hash],
 				vec![],
 				start.elapsed(),
 			);
 		});
 		self.db.read().key_value().flush().expect("DB flush failed.");
-		Ok(h)
+		Ok(hash)
 	}
 }
 
@@ -2406,6 +2444,8 @@ impl ProvingBlockChainClient for Client {
 	}
 }
 
+impl SnapshotClient for Client {}
+
 impl Drop for Client {
 	fn drop(&mut self) {
 		self.engine.stop();
@@ -2504,7 +2544,7 @@ mod tests {
 		use test_helpers::{generate_dummy_client_with_data};
 
 		let client = generate_dummy_client_with_data(2, 2, &[1.into(), 1.into()]);
-		let receipts = client.block_receipts(BlockId::Latest).unwrap();
+		let receipts = client.localized_block_receipts(BlockId::Latest).unwrap();
 
 		assert_eq!(receipts.len(), 2);
 		assert_eq!(receipts[0].transaction_index, 0);

ethcore/src/client/test_client.rs

@@ -686,7 +686,7 @@ impl BlockChainClient for TestBlockChainClient {
 		self.receipts.read().get(&id).cloned()
 	}
 
-	fn block_receipts(&self, _id: BlockId) -> Option<Vec<LocalizedReceipt>> {
+	fn localized_block_receipts(&self, _id: BlockId) -> Option<Vec<LocalizedReceipt>> {
 		Some(self.receipts.read().values().cloned().collect())
 	}
 
@@ -789,16 +789,14 @@ impl BlockChainClient for TestBlockChainClient {
 		None
 	}
 
-	fn encoded_block_receipts(&self, hash: &H256) -> Option<Bytes> {
+	fn block_receipts(&self, hash: &H256) -> Option<BlockReceipts> {
 		// starts with 'f' ?
 		if *hash > H256::from("f000000000000000000000000000000000000000000000000000000000000000") {
 			let receipt = BlockReceipts::new(vec![Receipt::new(
 				TransactionOutcome::StateRoot(H256::zero()),
 				U256::zero(),
 				vec![])]);
-			let mut rlp = RlpStream::new();
-			rlp.append(&receipt);
-			return Some(rlp.out());
+			return Some(receipt);
 		}
 		None
 	}

ethcore/src/client/traits.rs

@@ -20,7 +20,7 @@ use std::sync::Arc;
 use itertools::Itertools;
 
 use block::{OpenBlock, SealedBlock, ClosedBlock};
-use blockchain::TreeRoute;
+use blockchain::{BlockReceipts, TreeRoute};
 use client::Mode;
 use encoded;
 use vm::LastHashes;
@@ -282,7 +282,7 @@ pub trait BlockChainClient : Sync + Send + AccountData + BlockChain + CallContra
 	fn transaction_receipt(&self, id: TransactionId) -> Option<LocalizedReceipt>;
 
 	/// Get localized receipts for all transaction in given block.
-	fn block_receipts(&self, id: BlockId) -> Option<Vec<LocalizedReceipt>>;
+	fn localized_block_receipts(&self, id: BlockId) -> Option<Vec<LocalizedReceipt>>;
 
 	/// Get a tree route between `from` and `to`.
 	/// See `BlockChain::tree_route`.
@@ -294,12 +294,17 @@ pub trait BlockChainClient : Sync + Send + AccountData + BlockChain + CallContra
 	/// Get latest state node
 	fn state_data(&self, hash: &H256) -> Option<Bytes>;
 
-	/// Get raw block receipts data by block header hash.
-	fn encoded_block_receipts(&self, hash: &H256) -> Option<Bytes>;
+	/// Get block receipts data by block header hash.
+	fn block_receipts(&self, hash: &H256) -> Option<BlockReceipts>;
 
 	/// Get block queue information.
 	fn queue_info(&self) -> BlockQueueInfo;
 
+	/// Returns true if block queue is empty.
+	fn is_queue_empty(&self) -> bool {
+		self.queue_info().is_empty()
+	}
+
 	/// Clear block queue and abort all import activity.
 	fn clear_queue(&self);
 

ethcore/src/engines/authority_round/mod.rs

@@ -16,8 +16,8 @@
 
 //! A blockchain engine that supports a non-instant BFT proof-of-authority.
 
-use std::collections::{BTreeMap, HashSet};
-use std::fmt;
+use std::collections::{BTreeMap, BTreeSet, HashSet};
+use std::{cmp, fmt};
 use std::iter::FromIterator;
 use std::ops::Deref;
 use std::sync::atomic::{AtomicUsize, AtomicBool, Ordering as AtomicOrdering};
@@ -123,10 +123,10 @@ struct Step {
 }
 
 impl Step {
-	fn load(&self) -> usize { self.inner.load(AtomicOrdering::SeqCst) }
+	fn load(&self) -> u64 { self.inner.load(AtomicOrdering::SeqCst) as u64 }
 	fn duration_remaining(&self) -> Duration {
 		let now = unix_now();
-		let expected_seconds = (self.load() as u64)
+		let expected_seconds = self.load()
 			.checked_add(1)
 			.and_then(|ctr| ctr.checked_mul(self.duration as u64))
 			.map(Duration::from_secs);
@@ -162,8 +162,8 @@ impl Step {
 		}
 	}
 
-	fn check_future(&self, given: usize) -> Result<(), Option<OutOfBounds<u64>>> {
-		const REJECTED_STEP_DRIFT: usize = 4;
+	fn check_future(&self, given: u64) -> Result<(), Option<OutOfBounds<u64>>> {
+		const REJECTED_STEP_DRIFT: u64 = 4;
 
 		// Verify if the step is correct.
 		if given <= self.load() {
@@ -182,8 +182,8 @@ impl Step {
 			let d = self.duration as u64;
 			Err(Some(OutOfBounds {
 				min: None,
-				max: Some(d * current as u64),
-				found: d * given as u64,
+				max: Some(d * current),
+				found: d * given,
 			}))
 		} else {
 			Ok(())
@@ -192,8 +192,8 @@ impl Step {
 }
 
 // Chain scoring: total weight is sqrt(U256::max_value())*height - step
-fn calculate_score(parent_step: U256, current_step: U256, current_empty_steps: U256) -> U256 {
-	U256::from(U128::max_value()) + parent_step - current_step + current_empty_steps
+fn calculate_score(parent_step: u64, current_step: u64, current_empty_steps: usize) -> U256 {
+	U256::from(U128::max_value()) + U256::from(parent_step) - U256::from(current_step) + U256::from(current_empty_steps)
 }
 
 struct EpochManager {
@@ -284,13 +284,26 @@ impl EpochManager {
 /// A message broadcast by authorities when it's their turn to seal a block but there are no
 /// transactions. Other authorities accumulate these messages and later include them in the seal as
 /// proof.
-#[derive(Clone, Debug)]
+#[derive(Clone, Debug, PartialEq, Eq)]
 struct EmptyStep {
 	signature: H520,
-	step: usize,
+	step: u64,
 	parent_hash: H256,
 }
 
+impl PartialOrd for EmptyStep {
+	fn partial_cmp(&self, other: &Self) -> Option<cmp::Ordering> {
+		Some(self.cmp(other))
+	}
+}
+impl Ord for EmptyStep {
+	fn cmp(&self, other: &Self) -> cmp::Ordering {
+		self.step.cmp(&other.step)
+			.then_with(|| self.parent_hash.cmp(&other.parent_hash))
+			.then_with(|| self.signature.cmp(&other.signature))
+	}
+}
+
 impl EmptyStep {
 	fn from_sealed(sealed_empty_step: SealedEmptyStep, parent_hash: &H256) -> EmptyStep {
 		let signature = sealed_empty_step.signature;
@@ -353,7 +366,7 @@ pub fn empty_step_full_rlp(signature: &H520, empty_step_rlp: &[u8]) -> Vec<u8> {
 	s.out()
 }
 
-pub fn empty_step_rlp(step: usize, parent_hash: &H256) -> Vec<u8> {
+pub fn empty_step_rlp(step: u64, parent_hash: &H256) -> Vec<u8> {
 	let mut s = RlpStream::new_list(2);
 	s.append(&step).append(parent_hash);
 	s.out()
@@ -365,7 +378,7 @@ pub fn empty_step_rlp(step: usize, parent_hash: &H256) -> Vec<u8> {
 /// empty message is included.
 struct SealedEmptyStep {
 	signature: H520,
-	step: usize,
+	step: u64,
 }
 
 impl Encodable for SealedEmptyStep {
@@ -399,7 +412,7 @@ pub struct AuthorityRound {
 	validators: Box<ValidatorSet>,
 	validate_score_transition: u64,
 	validate_step_transition: u64,
-	empty_steps: Mutex<Vec<EmptyStep>>,
+	empty_steps: Mutex<BTreeSet<EmptyStep>>,
 	epoch_manager: Mutex<EpochManager>,
 	immediate_transitions: bool,
 	block_reward: U256,
@@ -494,7 +507,7 @@ fn header_expected_seal_fields(header: &Header, empty_steps_transition: u64) ->
 	}
 }
 
-fn header_step(header: &Header, empty_steps_transition: u64) -> Result<usize, ::rlp::DecoderError> {
+fn header_step(header: &Header, empty_steps_transition: u64) -> Result<u64, ::rlp::DecoderError> {
 	let expected_seal_fields = header_expected_seal_fields(header, empty_steps_transition);
 	Rlp::new(&header.seal().get(0).expect(
 		&format!("was either checked with verify_block_basic or is genesis; has {} fields; qed (Make sure the spec file has a correct genesis seal)", expected_seal_fields))).as_val()
@@ -533,17 +546,17 @@ fn header_empty_steps_signers(header: &Header, empty_steps_transition: u64) -> R
 	}
 }
 
-fn step_proposer(validators: &ValidatorSet, bh: &H256, step: usize) -> Address {
-	let proposer = validators.get(bh, step);
+fn step_proposer(validators: &ValidatorSet, bh: &H256, step: u64) -> Address {
+	let proposer = validators.get(bh, step as usize);
 	trace!(target: "engine", "Fetched proposer for step {}: {}", step, proposer);
 	proposer
 }
 
-fn is_step_proposer(validators: &ValidatorSet, bh: &H256, step: usize, address: &Address) -> bool {
+fn is_step_proposer(validators: &ValidatorSet, bh: &H256, step: u64, address: &Address) -> bool {
 	step_proposer(validators, bh, step) == *address
 }
 
-fn verify_timestamp(step: &Step, header_step: usize) -> Result<(), BlockError> {
+fn verify_timestamp(step: &Step, header_step: u64) -> Result<(), BlockError> {
 	match step.check_future(header_step) {
 		Err(None) => {
 			trace!(target: "engine", "verify_timestamp: block from the future");
@@ -564,7 +577,7 @@ fn verify_external(header: &Header, validators: &ValidatorSet, empty_steps_trans
 	let header_step = header_step(header, empty_steps_transition)?;
 
 	let proposer_signature = header_signature(header, empty_steps_transition)?;
-	let correct_proposer = validators.get(header.parent_hash(), header_step);
+	let correct_proposer = validators.get(header.parent_hash(), header_step as usize);
 	let is_invalid_proposer = *header.author() != correct_proposer || {
 		let empty_steps_rlp = if header.number() >= empty_steps_transition {
 			Some(header_empty_steps_raw(header))
@@ -634,13 +647,13 @@ impl AuthorityRound {
 			panic!("authority_round: step duration can't be zero")
 		}
 		let should_timeout = our_params.start_step.is_none();
-		let initial_step = our_params.start_step.unwrap_or_else(|| (unix_now().as_secs() / (our_params.step_duration as u64))) as usize;
+		let initial_step = our_params.start_step.unwrap_or_else(|| (unix_now().as_secs() / (our_params.step_duration as u64)));
 		let engine = Arc::new(
 			AuthorityRound {
 				transition_service: IoService::<()>::start()?,
 				step: Arc::new(PermissionedStep {
 					inner: Step {
-						inner: AtomicUsize::new(initial_step),
+						inner: AtomicUsize::new(initial_step as usize),
 						calibrate: our_params.start_step.is_none(),
 						duration: our_params.step_duration,
 					},
@@ -651,7 +664,7 @@ impl AuthorityRound {
 				validators: our_params.validators,
 				validate_score_transition: our_params.validate_score_transition,
 				validate_step_transition: our_params.validate_step_transition,
-				empty_steps: Mutex::new(Vec::new()),
+				empty_steps: Default::default(),
 				epoch_manager: Mutex::new(EpochManager::blank()),
 				immediate_transitions: our_params.immediate_transitions,
 				block_reward: our_params.block_reward,
@@ -699,22 +712,41 @@ impl AuthorityRound {
 		})
 	}
 
-	fn empty_steps(&self, from_step: U256, to_step: U256, parent_hash: H256) -> Vec<EmptyStep> {
-		self.empty_steps.lock().iter().filter(|e| {
-			U256::from(e.step) > from_step &&
-				U256::from(e.step) < to_step &&
-				e.parent_hash == parent_hash
-		}).cloned().collect()
+	fn empty_steps(&self, from_step: u64, to_step: u64, parent_hash: H256) -> Vec<EmptyStep> {
+		let from = EmptyStep {
+			step: from_step + 1,
+			parent_hash,
+			signature: Default::default(),
+		};
+		let to = EmptyStep {
+			step: to_step,
+			parent_hash: Default::default(),
+			signature: Default::default(),
+		};
+
+		if from >= to {
+			return vec![];
+		}
+
+		self.empty_steps.lock()
+			.range(from..to)
+			.filter(|e| e.parent_hash == parent_hash)
+			.cloned()
+			.collect()
 	}
 
-	fn clear_empty_steps(&self, step: U256) {
+	fn clear_empty_steps(&self, step: u64) {
 		// clear old `empty_steps` messages
-		self.empty_steps.lock().retain(|e| U256::from(e.step) > step);
+		let mut empty_steps = self.empty_steps.lock();
+		*empty_steps = empty_steps.split_off(&EmptyStep {
+			step: step + 1,
+			parent_hash: Default::default(),
+			signature: Default::default(),
+		});
 	}
 
 	fn handle_empty_step_message(&self, empty_step: EmptyStep) {
-		let mut empty_steps = self.empty_steps.lock();
-		empty_steps.push(empty_step);
+		self.empty_steps.lock().insert(empty_step);
 	}
 
 	fn generate_empty_step(&self, parent_hash: &H256) {
@@ -744,7 +776,7 @@ impl AuthorityRound {
 		}
 	}
 
-	fn report_skipped(&self, header: &Header, current_step: usize, parent_step: usize, validators: &ValidatorSet, set_number: u64) {
+	fn report_skipped(&self, header: &Header, current_step: u64, parent_step: u64, validators: &ValidatorSet, set_number: u64) {
 		// we're building on top of the genesis block so don't report any skipped steps
 		if header.number() == 1 {
 			return;
@@ -798,6 +830,7 @@ impl AuthorityRound {
 				}
 			};
 
+			let epoch_transition_hash = epoch_manager.epoch_transition_hash;
 			let ancestry_iter = ancestry.map(|header| {
 				let mut signers = vec![*header.author()];
 				signers.extend(parent_empty_steps_signers.drain(..));
@@ -815,10 +848,11 @@ impl AuthorityRound {
 					None
 				}
 			})
-				.while_some();
+				.while_some()
+				.take_while(|&(h, _)| h != epoch_transition_hash);
 
-			if let Err(_) = epoch_manager.finality_checker.build_ancestry_subchain(ancestry_iter) {
-				debug!(target: "engine", "inconsistent validator set within epoch");
+			if let Err(e) = epoch_manager.finality_checker.build_ancestry_subchain(ancestry_iter) {
+				debug!(target: "engine", "inconsistent validator set within epoch: {:?}", e);
 				return Vec::new();
 			}
 		}
@@ -935,12 +969,12 @@ impl Engine<EthereumMachine> for AuthorityRound {
 		let current_step = self.step.inner.load();
 
 		let current_empty_steps_len = if header.number() >= self.empty_steps_transition {
-			self.empty_steps(parent_step.into(), current_step.into(), parent.hash()).len()
+			self.empty_steps(parent_step, current_step, parent.hash()).len()
 		} else {
 			0
 		};
 
-		let score = calculate_score(parent_step.into(), current_step.into(), current_empty_steps_len.into());
+		let score = calculate_score(parent_step, current_step, current_empty_steps_len);
 		header.set_difficulty(score);
 	}
 
@@ -984,8 +1018,8 @@ impl Engine<EthereumMachine> for AuthorityRound {
 		}
 
 		let header = block.header();
-		let parent_step: U256 = header_step(parent, self.empty_steps_transition)
-			.expect("Header has been verified; qed").into();
+		let parent_step = header_step(parent, self.empty_steps_transition)
+			.expect("Header has been verified; qed");
 
 		let step = self.step.inner.load();
 
@@ -1020,7 +1054,7 @@ impl Engine<EthereumMachine> for AuthorityRound {
 		if is_step_proposer(&*validators, header.parent_hash(), step, header.author()) {
 			// this is guarded against by `can_propose` unless the block was signed
 			// on the same step (implies same key) and on a different node.
-			if parent_step == step.into() {
+			if parent_step == step {
 				warn!("Attempted to seal block on the same step as parent. Is this authority sealing with more than one node?");
 				return Seal::None;
 			}
@@ -1032,7 +1066,10 @@ impl Engine<EthereumMachine> for AuthorityRound {
 				block.transactions().is_empty() &&
 				empty_steps.len() < self.maximum_empty_steps {
 
-				self.generate_empty_step(header.parent_hash());
+				if self.step.can_propose.compare_and_swap(true, false, AtomicOrdering::SeqCst) {
+					self.generate_empty_step(header.parent_hash());
+				}
+
 				return Seal::None;
 			}
 
@@ -1056,7 +1093,7 @@ impl Engine<EthereumMachine> for AuthorityRound {
 					// report any skipped primaries between the parent block and
 					// the block we're sealing, unless we have empty steps enabled
 					if header.number() < self.empty_steps_transition {
-						self.report_skipped(header, step, u64::from(parent_step) as usize, &*validators, set_number);
+						self.report_skipped(header, step, parent_step, &*validators, set_number);
 					}
 
 					let mut fields = vec![
@@ -1448,9 +1485,9 @@ impl Engine<EthereumMachine> for AuthorityRound {
 		super::total_difficulty_fork_choice(new, current)
 	}
 
-	fn ancestry_actions(&self, block: &ExecutedBlock, ancestry: &mut Iterator<Item=ExtendedHeader>) -> Vec<AncestryAction> {
+	fn ancestry_actions(&self, header: &Header, ancestry: &mut Iterator<Item=ExtendedHeader>) -> Vec<AncestryAction> {
 		let finalized = self.build_finality(
-			block.header(),
+			header,
 			&mut ancestry.take_while(|e| !e.is_finalized).map(|e| e.header),
 		);
 
@@ -1591,12 +1628,12 @@ mod tests {
 
 		// Two validators.
 		// Spec starts with step 2.
-		header.set_difficulty(calculate_score(U256::from(0), U256::from(2), U256::zero()));
+		header.set_difficulty(calculate_score(0, 2, 0));
 		let signature = tap.sign(addr, Some("0".into()), header.bare_hash()).unwrap();
 		header.set_seal(vec![encode(&2usize), encode(&(&*signature as &[u8]))]);
 		assert!(engine.verify_block_family(&header, &parent_header).is_ok());
 		assert!(engine.verify_block_external(&header).is_err());
-		header.set_difficulty(calculate_score(U256::from(0), U256::from(1), U256::zero()));
+		header.set_difficulty(calculate_score(0, 1, 0));
 		let signature = tap.sign(addr, Some("0".into()), header.bare_hash()).unwrap();
 		header.set_seal(vec![encode(&1usize), encode(&(&*signature as &[u8]))]);
 		assert!(engine.verify_block_family(&header, &parent_header).is_ok());
@@ -1620,7 +1657,7 @@ mod tests {
 
 		// Two validators.
 		// Spec starts with step 2.
-		header.set_difficulty(calculate_score(U256::from(0), U256::from(1), U256::zero()));
+		header.set_difficulty(calculate_score(0, 1, 0));
 		let signature = tap.sign(addr, Some("0".into()), header.bare_hash()).unwrap();
 		header.set_seal(vec![encode(&1usize), encode(&(&*signature as &[u8]))]);
 		assert!(engine.verify_block_family(&header, &parent_header).is_ok());
@@ -1648,10 +1685,10 @@ mod tests {
 		// Two validators.
 		// Spec starts with step 2.
 		header.set_seal(vec![encode(&5usize), encode(&(&*signature as &[u8]))]);
-		header.set_difficulty(calculate_score(U256::from(4), U256::from(5), U256::zero()));
+		header.set_difficulty(calculate_score(4, 5, 0));
 		assert!(engine.verify_block_family(&header, &parent_header).is_ok());
 		header.set_seal(vec![encode(&3usize), encode(&(&*signature as &[u8]))]);
-		header.set_difficulty(calculate_score(U256::from(4), U256::from(3), U256::zero()));
+		header.set_difficulty(calculate_score(4, 3, 0));
 		assert!(engine.verify_block_family(&header, &parent_header).is_err());
 	}
 
@@ -1685,7 +1722,7 @@ mod tests {
 		parent_header.set_seal(vec![encode(&1usize)]);
 		parent_header.set_gas_limit("222222".parse::<U256>().unwrap());
 		let mut header: Header = Header::default();
-		header.set_difficulty(calculate_score(U256::from(1), U256::from(3), U256::zero()));
+		header.set_difficulty(calculate_score(1, 3, 0));
 		header.set_gas_limit("222222".parse::<U256>().unwrap());
 		header.set_seal(vec![encode(&3usize)]);
 
@@ -1799,14 +1836,14 @@ mod tests {
 		(spec, tap, accounts)
 	}
 
-	fn empty_step(engine: &EthEngine, step: usize, parent_hash: &H256) -> EmptyStep {
+	fn empty_step(engine: &EthEngine, step: u64, parent_hash: &H256) -> EmptyStep {
 		let empty_step_rlp = super::empty_step_rlp(step, parent_hash);
 		let signature = engine.sign(keccak(&empty_step_rlp)).unwrap().into();
 		let parent_hash = parent_hash.clone();
 		EmptyStep { step, signature, parent_hash }
 	}
 
-	fn sealed_empty_step(engine: &EthEngine, step: usize, parent_hash: &H256) -> SealedEmptyStep {
+	fn sealed_empty_step(engine: &EthEngine, step: u64, parent_hash: &H256) -> SealedEmptyStep {
 		let empty_step_rlp = super::empty_step_rlp(step, parent_hash);
 		let signature = engine.sign(keccak(&empty_step_rlp)).unwrap().into();
 		SealedEmptyStep { signature, step }
@@ -1842,6 +1879,11 @@ mod tests {
 
 		// we've received the message
 		assert!(notify.messages.read().contains(&empty_step_rlp));
+		let len = notify.messages.read().len();
+
+		// make sure that we don't generate empty step for the second time
+		assert_eq!(engine.generate_seal(b1.block(), &genesis_header), Seal::None);
+		assert_eq!(len, notify.messages.read().len());
 	}
 
 	#[test]
@@ -2056,7 +2098,7 @@ mod tests {
 		let empty_step3 = sealed_empty_step(engine, 3, &parent_header.hash());
 
 		let empty_steps = vec![empty_step2, empty_step3];
-		header.set_difficulty(calculate_score(U256::from(0), U256::from(4), U256::from(2)));
+		header.set_difficulty(calculate_score(0, 4, 2));
 		let signature = tap.sign(addr1, Some("1".into()), header.bare_hash()).unwrap();
 		header.set_seal(vec![
 			encode(&4usize),
@@ -2171,4 +2213,52 @@ mod tests {
 			BTreeMap::default(),
 		);
 	}
+
+	#[test]
+	fn test_empty_steps() {
+		let last_benign = Arc::new(AtomicUsize::new(0));
+		let params = AuthorityRoundParams {
+			step_duration: 4,
+			start_step: Some(1),
+			validators: Box::new(TestSet::new(Default::default(), last_benign.clone())),
+			validate_score_transition: 0,
+			validate_step_transition: 0,
+			immediate_transitions: true,
+			maximum_uncle_count_transition: 0,
+			maximum_uncle_count: 0,
+			empty_steps_transition: 0,
+			maximum_empty_steps: 10,
+			block_reward: Default::default(),
+			block_reward_contract_transition: 0,
+			block_reward_contract: Default::default(),
+		};
+
+		let mut c_params = ::spec::CommonParams::default();
+		c_params.gas_limit_bound_divisor = 5.into();
+		let machine = ::machine::EthereumMachine::regular(c_params, Default::default());
+		let engine = AuthorityRound::new(params, machine).unwrap();
+
+
+		let parent_hash: H256 = 1.into();
+		let signature = H520::default();
+		let step = |step: u64| EmptyStep {
+			step,
+			parent_hash,
+			signature,
+		};
+
+		engine.handle_empty_step_message(step(1));
+		engine.handle_empty_step_message(step(3));
+		engine.handle_empty_step_message(step(2));
+		engine.handle_empty_step_message(step(1));
+
+		assert_eq!(engine.empty_steps(0, 4, parent_hash), vec![step(1), step(2), step(3)]);
+		assert_eq!(engine.empty_steps(2, 3, parent_hash), vec![]);
+		assert_eq!(engine.empty_steps(2, 4, parent_hash), vec![step(3)]);
+
+		engine.clear_empty_steps(2);
+
+		assert_eq!(engine.empty_steps(0, 3, parent_hash), vec![]);
+		assert_eq!(engine.empty_steps(0, 4, parent_hash), vec![step(3)]);
+	}
 }

ethcore/src/engines/mod.rs

@@ -254,6 +254,9 @@ pub trait Engine<M: Machine>: Sync + Send {
 	/// The number of generations back that uncles can be.
 	fn maximum_uncle_age(&self) -> usize { 6 }
 
+	/// Optional maximum gas limit.
+	fn maximum_gas_limit(&self) -> Option<U256> { None }
+
 	/// Block transformation functions, before the transactions.
 	/// `epoch_begin` set to true if this block kicks off an epoch.
 	fn on_new_block(
@@ -426,7 +429,7 @@ pub trait Engine<M: Machine>: Sync + Send {
 
 	/// Gather all ancestry actions. Called at the last stage when a block is committed. The Engine must guarantee that
 	/// the ancestry exists.
-	fn ancestry_actions(&self, _block: &M::LiveBlock, _ancestry: &mut Iterator<Item=M::ExtendedHeader>) -> Vec<AncestryAction> {
+	fn ancestry_actions(&self, _header: &M::Header, _ancestry: &mut Iterator<Item=M::ExtendedHeader>) -> Vec<AncestryAction> {
 		Vec::new()
 	}
 

ethcore/src/ethereum/ethash.rs

@@ -222,6 +222,8 @@ impl Engine<EthereumMachine> for Arc<Ethash> {
 
 	fn maximum_uncle_count(&self, _block: BlockNumber) -> usize { 2 }
 
+	fn maximum_gas_limit(&self) -> Option<U256> { Some(0x7fff_ffff_ffff_ffffu64.into()) }
+
 	fn populate_from_parent(&self, header: &mut Header, parent: &Header) {
 		let difficulty = self.calculate_difficulty(header, parent);
 		header.set_difficulty(difficulty);
@@ -322,10 +324,6 @@ impl Engine<EthereumMachine> for Arc<Ethash> {
 			return Err(From::from(BlockError::InvalidProofOfWork(OutOfBounds { min: Some(header.difficulty().clone()), max: None, found: difficulty })));
 		}
 
-		if header.gas_limit() > &0x7fffffffffffffffu64.into() {
-			return Err(From::from(BlockError::InvalidGasLimit(OutOfBounds { min: None, max: Some(0x7fffffffffffffffu64.into()), found: header.gas_limit().clone() })));
-		}
-
 		Ok(())
 	}
 

ethcore/src/json_tests/chain.rs

@@ -23,8 +23,8 @@ use miner::Miner;
 use io::IoChannel;
 use test_helpers;
 use verification::queue::kind::blocks::Unverified;
+use verification::VerifierType;
 use super::SKIP_TEST_STATE;
-
 use super::HookType;
 
 /// Run chain jsontests on a given folder.
@@ -84,6 +84,10 @@ pub fn json_chain_test<H: FnMut(&str, HookType)>(json_data: &[u8], start_stop_ho
 			{
 				let db = test_helpers::new_db();
 				let mut config = ClientConfig::default();
+				if ethjson::blockchain::Engine::NoProof == blockchain.engine {
+					config.verifier_type = VerifierType::CanonNoSeal;
+					config.check_seal = false;
+				}
 				config.history = 8;
 				let client = Client::new(
 					config,
@@ -184,6 +188,7 @@ mod block_tests {
 	declare_test!{BlockchainTests_GeneralStateTest_stZeroCallsTest, "BlockchainTests/GeneralStateTests/stZeroCallsTest/"}
 	declare_test!{BlockchainTests_GeneralStateTest_stZeroKnowledge, "BlockchainTests/GeneralStateTests/stZeroKnowledge/"}
 	declare_test!{BlockchainTests_GeneralStateTest_stZeroKnowledge2, "BlockchainTests/GeneralStateTests/stZeroKnowledge2/"}
+	declare_test!{BlockchainTests_GeneralStateTest_stSStoreTest, "BlockchainTests/GeneralStateTests/stSStoreTest/"}
 
 	declare_test!{BlockchainTests_TransitionTests_bcEIP158ToByzantium, "BlockchainTests/TransitionTests/bcEIP158ToByzantium/"}
 	declare_test!{BlockchainTests_TransitionTests_bcFrontierToHomestead, "BlockchainTests/TransitionTests/bcFrontierToHomestead/"}

ethcore/src/json_tests/state.rs

@@ -177,6 +177,7 @@ mod state_tests {
 	declare_test!{GeneralStateTest_stZeroCallsRevert, "GeneralStateTests/stZeroCallsRevert/"}
 	declare_test!{GeneralStateTest_stZeroCallsTest, "GeneralStateTests/stZeroCallsTest/"}
 	declare_test!{GeneralStateTest_stZeroKnowledge, "GeneralStateTests/stZeroKnowledge/"}
+	declare_test!{GeneralStateTest_stSStoreTest, "GeneralStateTests/stSStoreTest/"}
 
 	// Attempts to send a transaction that requires more than current balance:
 	// Tx:

ethcore/src/lib.rs

@@ -139,6 +139,9 @@ extern crate trace_time;
 #[cfg_attr(test, macro_use)]
 extern crate evm;
 
+#[cfg(test)]
+extern crate env_logger;
+
 pub extern crate ethstore;
 
 #[macro_use]

ethcore/src/machine.rs

@@ -371,11 +371,11 @@ impl EthereumMachine {
 	}
 
 	/// Does verification of the transaction against the parent state.
-	pub fn verify_transaction<C: BlockInfo + CallContract>(&self, t: &SignedTransaction, header: &Header, client: &C)
+	pub fn verify_transaction<C: BlockInfo + CallContract>(&self, t: &SignedTransaction, parent: &Header, client: &C)
 		-> Result<(), transaction::Error>
 	{
 		if let Some(ref filter) = self.tx_filter.as_ref() {
-			if !filter.transaction_allowed(header.parent_hash(), header.number(), t, client) {
+			if !filter.transaction_allowed(&parent.hash(), parent.number() + 1, t, client) {
 				return Err(transaction::Error::NotAllowed.into())
 			}
 		}

ethcore/src/miner/miner.rs

@@ -576,10 +576,9 @@ impl Miner {
 		trace!(target: "miner", "requires_reseal: sealing enabled");
 
 		// Disable sealing if there were no requests for SEALING_TIMEOUT_IN_BLOCKS
-		let had_requests = sealing.last_request.map(|last_request| {
-			best_block > last_request
-				&& best_block - last_request <= SEALING_TIMEOUT_IN_BLOCKS
-		}).unwrap_or(false);
+		let had_requests = sealing.last_request.map(|last_request|
+			best_block.saturating_sub(last_request) <= SEALING_TIMEOUT_IN_BLOCKS
+		).unwrap_or(false);
 
 		// keep sealing enabled if any of the conditions is met
 		let sealing_enabled = self.forced_sealing()
@@ -1394,6 +1393,33 @@ mod tests {
 		assert_eq!(miner.prepare_pending_block(&client), BlockPreparationStatus::NotPrepared);
 	}
 
+	#[test]
+	fn should_not_return_stale_work_packages() {
+		// given
+		let client = TestBlockChainClient::default();
+		let miner = miner();
+
+		// initial work package should create the pending block
+		let res = miner.work_package(&client);
+		assert_eq!(res.unwrap().1, 1);
+		// This should be true, since there were some requests.
+		assert_eq!(miner.requires_reseal(0), true);
+
+		// when new block is imported
+		let client = generate_dummy_client(2);
+		let imported = [0.into()];
+		let empty = &[];
+		miner.chain_new_blocks(&*client, &imported, empty, &imported, empty, false);
+
+		// then
+		// This should be false, because it's too early.
+		assert_eq!(miner.requires_reseal(2), false);
+		// but still work package should be ready
+		let res = miner.work_package(&*client);
+		assert_eq!(res.unwrap().1, 3);
+		assert_eq!(miner.prepare_pending_block(&*client), BlockPreparationStatus::NotPrepared);
+	}
+
 	#[test]
 	fn should_not_use_pending_block_if_best_block_is_higher() {
 		// given

ethcore/src/snapshot/error.rs

@@ -65,6 +65,8 @@ pub enum Error {
 	BadEpochProof(u64),
 	/// Wrong chunk format.
 	WrongChunkFormat(String),
+	/// Unlinked ancient block chain
+	UnlinkedAncientBlockChain,
 }
 
 impl fmt::Display for Error {
@@ -91,6 +93,7 @@ impl fmt::Display for Error {
 			Error::SnapshotsUnsupported => write!(f, "Snapshots unsupported by consensus engine."),
 			Error::BadEpochProof(i) => write!(f, "Bad epoch proof for transition to epoch {}", i),
 			Error::WrongChunkFormat(ref msg) => write!(f, "Wrong chunk format: {}", msg),
+			Error::UnlinkedAncientBlockChain => write!(f, "Unlinked ancient blocks chain"),
 		}
 	}
 }

ethcore/src/snapshot/mod.rs

@@ -56,7 +56,7 @@ use rand::{Rng, OsRng};
 pub use self::error::Error;
 
 pub use self::consensus::*;
-pub use self::service::{Service, DatabaseRestore};
+pub use self::service::{SnapshotClient, Service, DatabaseRestore};
 pub use self::traits::SnapshotService;
 pub use self::watcher::Watcher;
 pub use types::snapshot_manifest::ManifestData;

ethcore/src/snapshot/service.rs

@@ -22,12 +22,13 @@ use std::fs::{self, File};
 use std::path::PathBuf;
 use std::sync::Arc;
 use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
+use std::cmp;
 
 use super::{ManifestData, StateRebuilder, Rebuilder, RestorationStatus, SnapshotService, MAX_CHUNK_SIZE};
 use super::io::{SnapshotReader, LooseReader, SnapshotWriter, LooseWriter};
 
 use blockchain::{BlockChain, BlockChainDB, BlockChainDBHandler};
-use client::{Client, ChainInfo, ClientIoMessage};
+use client::{BlockInfo, BlockChainClient, Client, ChainInfo, ClientIoMessage};
 use engines::EthEngine;
 use error::{Error, ErrorKind as SnapshotErrorKind};
 use snapshot::{Error as SnapshotError};
@@ -40,6 +41,7 @@ use ethereum_types::H256;
 use parking_lot::{Mutex, RwLock, RwLockReadGuard};
 use bytes::Bytes;
 use journaldb::Algorithm;
+use kvdb::DBTransaction;
 use snappy;
 
 /// Helper for removing directories in case of error.
@@ -203,6 +205,9 @@ impl Restoration {
 /// Type alias for client io channel.
 pub type Channel = IoChannel<ClientIoMessage>;
 
+/// Trait alias for the Client Service used
+pub trait SnapshotClient: BlockChainClient + BlockInfo + DatabaseRestore {}
+
 /// Snapshot service parameters.
 pub struct ServiceParams {
 	/// The consensus engine this is built on.
@@ -219,7 +224,7 @@ pub struct ServiceParams {
 	/// Usually "<chain hash>/snapshot"
 	pub snapshot_root: PathBuf,
 	/// A handle for database restoration.
-	pub db_restore: Arc<DatabaseRestore>,
+	pub client: Arc<SnapshotClient>,
 }
 
 /// `SnapshotService` implementation.
@@ -236,7 +241,7 @@ pub struct Service {
 	genesis_block: Bytes,
 	state_chunks: AtomicUsize,
 	block_chunks: AtomicUsize,
-	db_restore: Arc<DatabaseRestore>,
+	client: Arc<SnapshotClient>,
 	progress: super::Progress,
 	taking_snapshot: AtomicBool,
 	restoring_snapshot: AtomicBool,
@@ -257,7 +262,7 @@ impl Service {
 			genesis_block: params.genesis_block,
 			state_chunks: AtomicUsize::new(0),
 			block_chunks: AtomicUsize::new(0),
-			db_restore: params.db_restore,
+			client: params.client,
 			progress: Default::default(),
 			taking_snapshot: AtomicBool::new(false),
 			restoring_snapshot: AtomicBool::new(false),
@@ -334,12 +339,110 @@ impl Service {
 
 	// replace one the client's database with our own.
 	fn replace_client_db(&self) -> Result<(), Error> {
-		let our_db = self.restoration_db();
+		let migrated_blocks = self.migrate_blocks()?;
+		trace!(target: "snapshot", "Migrated {} ancient blocks", migrated_blocks);
 
-		self.db_restore.restore_db(&*our_db.to_string_lossy())?;
+		let rest_db = self.restoration_db();
+		self.client.restore_db(&*rest_db.to_string_lossy())?;
 		Ok(())
 	}
 
+	// Migrate the blocks in the current DB into the new chain
+	fn migrate_blocks(&self) -> Result<usize, Error> {
+		// Count the number of migrated blocks
+		let mut count = 0;
+		let rest_db = self.restoration_db();
+
+		let cur_chain_info = self.client.chain_info();
+
+		let next_db = self.restoration_db_handler.open(&rest_db)?;
+		let next_chain = BlockChain::new(Default::default(), &[], next_db.clone());
+		let next_chain_info = next_chain.chain_info();
+
+		// The old database looks like this:
+		// [genesis, best_ancient_block] ... [first_block, best_block]
+		// If we are fully synced neither `best_ancient_block` nor `first_block` is set, and we can assume that the whole range from [genesis, best_block] is imported.
+		// The new database only contains the tip of the chain ([first_block, best_block]),
+		// so the useful set of blocks is defined as:
+		// [0 ... min(new.first_block, best_ancient_block or best_block)]
+		let find_range = || -> Option<(H256, H256)> {
+			let next_available_from = next_chain_info.first_block_number?;
+			let cur_available_to = cur_chain_info.ancient_block_number.unwrap_or(cur_chain_info.best_block_number);
+
+			let highest_block_num = cmp::min(next_available_from.saturating_sub(1), cur_available_to);
+
+			if highest_block_num == 0 {
+				return None;
+			}
+
+			trace!(target: "snapshot", "Trying to import ancient blocks until {}", highest_block_num);
+
+			// Here we start from the highest block number and go backward to 0,
+			// thus starting at `highest_block_num` and targetting `0`.
+			let target_hash = self.client.block_hash(BlockId::Number(0))?;
+			let start_hash = self.client.block_hash(BlockId::Number(highest_block_num))?;
+
+			Some((start_hash, target_hash))
+		};
+
+		let (start_hash, target_hash) = match find_range() {
+			Some(x) => x,
+			None => return Ok(0),
+		};
+
+		let mut batch = DBTransaction::new();
+		let mut parent_hash = start_hash;
+		while parent_hash != target_hash {
+			// Early return if restoration is aborted
+			if !self.restoring_snapshot.load(Ordering::SeqCst) {
+				return Ok(count);
+			}
+
+			let block = self.client.block(BlockId::Hash(parent_hash)).ok_or(::snapshot::error::Error::UnlinkedAncientBlockChain)?;
+			parent_hash = block.parent_hash();
+
+			let block_number = block.number();
+			let block_receipts = self.client.block_receipts(&block.hash());
+			let parent_total_difficulty = self.client.block_total_difficulty(BlockId::Hash(parent_hash));
+
+			match (block_receipts, parent_total_difficulty) {
+				(Some(block_receipts), Some(parent_total_difficulty)) => {
+					let block_receipts = block_receipts.receipts;
+
+					next_chain.insert_unordered_block(&mut batch, block, block_receipts, Some(parent_total_difficulty), false, true);
+					count += 1;
+				},
+				_ => break,
+			}
+
+			// Writting changes to DB and logging every now and then
+			if block_number % 1_000 == 0 {
+				next_db.key_value().write_buffered(batch);
+				next_chain.commit();
+				next_db.key_value().flush().expect("DB flush failed.");
+				batch = DBTransaction::new();
+			}
+
+			if block_number % 10_000 == 0 {
+				trace!(target: "snapshot", "Block restoration at #{}", block_number);
+			}
+		}
+
+		// Final commit to the DB
+		next_db.key_value().write_buffered(batch);
+		next_chain.commit();
+		next_db.key_value().flush().expect("DB flush failed.");
+
+		// We couldn't reach the targeted hash
+		if parent_hash != target_hash {
+			return Err(::snapshot::error::Error::UnlinkedAncientBlockChain.into());
+		}
+
+		// Update best ancient block in the Next Chain
+		next_chain.update_best_ancient_block(&start_hash);
+		Ok(count)
+	}
+
 	/// Get a reference to the snapshot reader.
 	pub fn reader(&self) -> RwLockReadGuard<Option<LooseReader>> {
 		self.reader.read()
@@ -480,12 +583,16 @@ impl Service {
 		// Import previous chunks, continue if it fails
 		self.import_prev_chunks(&mut res, manifest).ok();
 
-		*self.status.lock() = RestorationStatus::Ongoing {
-			state_chunks: state_chunks as u32,
-			block_chunks: block_chunks as u32,
-			state_chunks_done: self.state_chunks.load(Ordering::SeqCst) as u32,
-			block_chunks_done: self.block_chunks.load(Ordering::SeqCst) as u32,
-		};
+		// It could be that the restoration failed or completed in the meanwhile
+		let mut restoration_status = self.status.lock();
+		if let RestorationStatus::Initializing { .. } = *restoration_status {
+			*restoration_status = RestorationStatus::Ongoing {
+				state_chunks: state_chunks as u32,
+				block_chunks: block_chunks as u32,
+				state_chunks_done: self.state_chunks.load(Ordering::SeqCst) as u32,
+				block_chunks_done: self.block_chunks.load(Ordering::SeqCst) as u32,
+			};
+		}
 
 		Ok(())
 	}
@@ -752,26 +859,19 @@ impl Drop for Service {
 
 #[cfg(test)]
 mod tests {
-	use std::sync::Arc;
 	use client::ClientIoMessage;
 	use io::{IoService};
 	use spec::Spec;
 	use journaldb::Algorithm;
-	use error::Error;
 	use snapshot::{ManifestData, RestorationStatus, SnapshotService};
 	use super::*;
 	use tempdir::TempDir;
-	use test_helpers::restoration_db_handler;
-
-	struct NoopDBRestore;
-	impl DatabaseRestore for NoopDBRestore {
-		fn restore_db(&self, _new_db: &str) -> Result<(), Error> {
-			Ok(())
-		}
-	}
+	use test_helpers::{generate_dummy_client_with_spec_and_data, restoration_db_handler};
 
 	#[test]
 	fn sends_async_messages() {
+		let gas_prices = vec![1.into(), 2.into(), 3.into(), 999.into()];
+		let client = generate_dummy_client_with_spec_and_data(Spec::new_null, 400, 5, &gas_prices);
 		let service = IoService::<ClientIoMessage>::start().unwrap();
 		let spec = Spec::new_test();
 
@@ -785,7 +885,7 @@ mod tests {
 			pruning: Algorithm::Archive,
 			channel: service.channel(),
 			snapshot_root: dir,
-			db_restore: Arc::new(NoopDBRestore),
+			client: client,
 		};
 
 		let service = Service::new(snapshot_params).unwrap();

ethcore/src/snapshot/tests/service.rs

@@ -16,26 +16,23 @@
 
 //! Tests for the snapshot service.
 
+use std::fs;
 use std::sync::Arc;
 
 use tempdir::TempDir;
-use client::{Client, BlockInfo};
+use blockchain::BlockProvider;
+use client::{Client, ClientConfig, ImportBlock, BlockInfo};
 use ids::BlockId;
+use snapshot::io::{PackedReader, PackedWriter, SnapshotReader, SnapshotWriter};
 use snapshot::service::{Service, ServiceParams};
-use snapshot::{self, ManifestData, SnapshotService};
+use snapshot::{chunk_state, chunk_secondary, ManifestData, Progress, SnapshotService, RestorationStatus};
 use spec::Spec;
-use test_helpers::{generate_dummy_client_with_spec_and_data, restoration_db_handler};
+use test_helpers::{new_db, new_temp_db, generate_dummy_client_with_spec_and_data, restoration_db_handler};
 
+use parking_lot::Mutex;
 use io::IoChannel;
 use kvdb_rocksdb::DatabaseConfig;
-
-struct NoopDBRestore;
-
-impl snapshot::DatabaseRestore for NoopDBRestore {
-	fn restore_db(&self, _new_db: &str) -> Result<(), ::error::Error> {
-		Ok(())
-	}
-}
+use verification::queue::kind::blocks::Unverified;
 
 #[test]
 fn restored_is_equivalent() {
@@ -46,7 +43,6 @@ fn restored_is_equivalent() {
 	const TX_PER: usize = 5;
 
 	let gas_prices = vec![1.into(), 2.into(), 3.into(), 999.into()];
-
 	let client = generate_dummy_client_with_spec_and_data(Spec::new_null, NUM_BLOCKS, TX_PER, &gas_prices);
 
 	let tempdir = TempDir::new("").unwrap();
@@ -73,7 +69,7 @@ fn restored_is_equivalent() {
 		pruning: ::journaldb::Algorithm::Archive,
 		channel: IoChannel::disconnected(),
 		snapshot_root: path,
-		db_restore: client2.clone(),
+		client: client2.clone(),
 	};
 
 	let service = Service::new(service_params).unwrap();
@@ -94,7 +90,7 @@ fn restored_is_equivalent() {
 		service.feed_block_chunk(hash, &chunk);
 	}
 
-	assert_eq!(service.status(), ::snapshot::RestorationStatus::Inactive);
+	assert_eq!(service.status(), RestorationStatus::Inactive);
 
 	for x in 0..NUM_BLOCKS {
 		let block1 = client.block(BlockId::Number(x as u64)).unwrap();
@@ -106,6 +102,9 @@ fn restored_is_equivalent() {
 
 #[test]
 fn guards_delete_folders() {
+	let gas_prices = vec![1.into(), 2.into(), 3.into(), 999.into()];
+	let client = generate_dummy_client_with_spec_and_data(Spec::new_null, 400, 5, &gas_prices);
+
 	let spec = Spec::new_null();
 	let tempdir = TempDir::new("").unwrap();
 	let service_params = ServiceParams {
@@ -115,7 +114,7 @@ fn guards_delete_folders() {
 		pruning: ::journaldb::Algorithm::Archive,
 		channel: IoChannel::disconnected(),
 		snapshot_root: tempdir.path().to_owned(),
-		db_restore: Arc::new(NoopDBRestore),
+		client: client,
 	};
 
 	let service = Service::new(service_params).unwrap();
@@ -146,3 +145,201 @@ fn guards_delete_folders() {
 	assert!(!path.join("db").exists());
 	assert!(path.join("temp").exists());
 }
+
+#[test]
+fn keep_ancient_blocks() {
+	::env_logger::init().ok();
+
+	// Test variables
+	const NUM_BLOCKS: u64 = 500;
+	const NUM_SNAPSHOT_BLOCKS: u64 = 300;
+	const SNAPSHOT_MODE: ::snapshot::PowSnapshot = ::snapshot::PowSnapshot { blocks: NUM_SNAPSHOT_BLOCKS, max_restore_blocks: NUM_SNAPSHOT_BLOCKS };
+
+	// Temporary folders
+	let tempdir = TempDir::new("").unwrap();
+	let snapshot_path = tempdir.path().join("SNAP");
+
+	// Generate blocks
+	let gas_prices = vec![1.into(), 2.into(), 3.into(), 999.into()];
+	let spec_f = Spec::new_null;
+	let spec = spec_f();
+	let client = generate_dummy_client_with_spec_and_data(spec_f, NUM_BLOCKS as u32, 5, &gas_prices);
+
+	let bc = client.chain();
+
+	// Create the Snapshot
+	let best_hash = bc.best_block_hash();
+	let writer = Mutex::new(PackedWriter::new(&snapshot_path).unwrap());
+	let block_hashes = chunk_secondary(
+		Box::new(SNAPSHOT_MODE),
+		&bc,
+		best_hash,
+		&writer,
+		&Progress::default()
+	).unwrap();
+	let state_db = client.state_db().journal_db().boxed_clone();
+	let start_header = bc.block_header_data(&best_hash).unwrap();
+	let state_root = start_header.state_root();
+	let state_hashes = chunk_state(
+		state_db.as_hashdb(),
+		&state_root,
+		&writer,
+		&Progress::default(),
+		None
+	).unwrap();
+
+	let manifest = ::snapshot::ManifestData {
+		version: 2,
+		state_hashes: state_hashes,
+		state_root: state_root,
+		block_hashes: block_hashes,
+		block_number: NUM_BLOCKS,
+		block_hash: best_hash,
+	};
+
+	writer.into_inner().finish(manifest.clone()).unwrap();
+
+	// Initialize the Client
+	let db_config = DatabaseConfig::with_columns(::db::NUM_COLUMNS);
+	let client_db = new_temp_db(&tempdir.path());
+	let client2 = Client::new(
+		ClientConfig::default(),
+		&spec,
+		client_db,
+		Arc::new(::miner::Miner::new_for_tests(&spec, None)),
+		IoChannel::disconnected(),
+	).unwrap();
+
+	// Add some ancient blocks
+	for block_number in 1..50 {
+		let block_hash = bc.block_hash(block_number).unwrap();
+		let block = bc.block(&block_hash).unwrap();
+		client2.import_block(Unverified::from_rlp(block.into_inner()).unwrap()).unwrap();
+	}
+
+	client2.import_verified_blocks();
+	client2.flush_queue();
+
+	// Restore the Snapshot
+	let reader = PackedReader::new(&snapshot_path).unwrap().unwrap();
+	let service_params = ServiceParams {
+		engine: spec.engine.clone(),
+		genesis_block: spec.genesis_block(),
+		restoration_db_handler: restoration_db_handler(db_config),
+		pruning: ::journaldb::Algorithm::Archive,
+		channel: IoChannel::disconnected(),
+		snapshot_root: tempdir.path().to_owned(),
+		client: client2.clone(),
+	};
+	let service = Service::new(service_params).unwrap();
+	service.init_restore(manifest.clone(), false).unwrap();
+
+	for hash in &manifest.block_hashes {
+		let chunk = reader.chunk(*hash).unwrap();
+		service.feed_block_chunk(*hash, &chunk);
+	}
+
+	for hash in &manifest.state_hashes {
+		let chunk = reader.chunk(*hash).unwrap();
+		service.feed_state_chunk(*hash, &chunk);
+	}
+
+	match service.status() {
+		RestorationStatus::Inactive => (),
+		RestorationStatus::Failed => panic!("Snapshot Restoration has failed."),
+		RestorationStatus::Ongoing { .. } => panic!("Snapshot Restoration should be done."),
+		_ => panic!("Invalid Snapshot Service status."),
+	}
+
+	// Check that the latest block number is the right one
+	assert_eq!(client2.block(BlockId::Latest).unwrap().number(), NUM_BLOCKS as u64);
+
+	// Check that we have blocks in [NUM_BLOCKS - NUM_SNAPSHOT_BLOCKS + 1 ; NUM_BLOCKS]
+	// but none before
+	assert!(client2.block(BlockId::Number(NUM_BLOCKS - NUM_SNAPSHOT_BLOCKS + 1)).is_some());
+	assert!(client2.block(BlockId::Number(100)).is_none());
+
+	// Check that the first 50 blocks have been migrated
+	for block_number in 1..49 {
+		assert!(client2.block(BlockId::Number(block_number)).is_some());
+	}
+}
+
+#[test]
+fn recover_aborted_recovery() {
+	::env_logger::init().ok();
+
+	const NUM_BLOCKS: u32 = 400;
+	let gas_prices = vec![1.into(), 2.into(), 3.into(), 999.into()];
+	let client = generate_dummy_client_with_spec_and_data(Spec::new_null, NUM_BLOCKS, 5, &gas_prices);
+
+	let spec = Spec::new_null();
+	let tempdir = TempDir::new("").unwrap();
+	let db_config = DatabaseConfig::with_columns(::db::NUM_COLUMNS);
+	let client_db = new_db();
+	let client2 = Client::new(
+		Default::default(),
+		&spec,
+		client_db,
+		Arc::new(::miner::Miner::new_for_tests(&spec, None)),
+		IoChannel::disconnected(),
+	).unwrap();
+	let service_params = ServiceParams {
+		engine: spec.engine.clone(),
+		genesis_block: spec.genesis_block(),
+		restoration_db_handler: restoration_db_handler(db_config),
+		pruning: ::journaldb::Algorithm::Archive,
+		channel: IoChannel::disconnected(),
+		snapshot_root: tempdir.path().to_owned(),
+		client: client2.clone(),
+	};
+
+	let service = Service::new(service_params).unwrap();
+	service.take_snapshot(&client, NUM_BLOCKS as u64).unwrap();
+
+	let manifest = service.manifest().unwrap();
+	service.init_restore(manifest.clone(), true).unwrap();
+
+	// Restore only the state chunks
+	for hash in &manifest.state_hashes {
+		let chunk = service.chunk(*hash).unwrap();
+		service.feed_state_chunk(*hash, &chunk);
+	}
+
+	match service.status() {
+		RestorationStatus::Ongoing { block_chunks_done, state_chunks_done, .. } => {
+			assert_eq!(state_chunks_done, manifest.state_hashes.len() as u32);
+			assert_eq!(block_chunks_done, 0);
+		},
+		e => panic!("Snapshot restoration must be ongoing ; {:?}", e),
+	}
+
+	// Abort the restore...
+	service.abort_restore();
+
+	// And try again!
+	service.init_restore(manifest.clone(), true).unwrap();
+
+	match service.status() {
+		RestorationStatus::Ongoing { block_chunks_done, state_chunks_done, .. } => {
+			assert_eq!(state_chunks_done, manifest.state_hashes.len() as u32);
+			assert_eq!(block_chunks_done, 0);
+		},
+		e => panic!("Snapshot restoration must be ongoing ; {:?}", e),
+	}
+
+	// Remove the snapshot directory, and restart the restoration
+	// It shouldn't have restored any previous blocks
+	fs::remove_dir_all(tempdir.path()).unwrap();
+
+	// And try again!
+	service.init_restore(manifest.clone(), true).unwrap();
+
+	match service.status() {
+		RestorationStatus::Ongoing { block_chunks_done, state_chunks_done, .. } => {
+			assert_eq!(block_chunks_done, 0);
+			assert_eq!(state_chunks_done, 0);
+		},
+		_ => panic!("Snapshot restoration must be ongoing"),
+	}
+}

ethcore/src/test_helpers.rs

@@ -41,7 +41,7 @@ use transaction::{Action, Transaction, SignedTransaction};
 use views::BlockView;
 use blooms_db;
 use kvdb::KeyValueDB;
-use kvdb_rocksdb;
+use kvdb_rocksdb::{self, Database, DatabaseConfig};
 use tempdir::TempDir;
 use verification::queue::kind::blocks::Unverified;
 use encoded;
@@ -263,30 +263,30 @@ pub fn get_test_client_with_blocks(blocks: Vec<Bytes>) -> Arc<Client> {
 	client
 }
 
+struct TestBlockChainDB {
+	_blooms_dir: TempDir,
+	_trace_blooms_dir: TempDir,
+	blooms: blooms_db::Database,
+	trace_blooms: blooms_db::Database,
+	key_value: Arc<KeyValueDB>,
+}
+
+impl BlockChainDB for TestBlockChainDB {
+	fn key_value(&self) -> &Arc<KeyValueDB> {
+		&self.key_value
+	}
+
+	fn blooms(&self) -> &blooms_db::Database {
+		&self.blooms
+	}
+
+	fn trace_blooms(&self) -> &blooms_db::Database {
+		&self.trace_blooms
+	}
+}
+
 /// Creates new test instance of `BlockChainDB`
 pub fn new_db() -> Arc<BlockChainDB> {
-	struct TestBlockChainDB {
-		_blooms_dir: TempDir,
-		_trace_blooms_dir: TempDir,
-		blooms: blooms_db::Database,
-		trace_blooms: blooms_db::Database,
-		key_value: Arc<KeyValueDB>,
-	}
-
-	impl BlockChainDB for TestBlockChainDB {
-		fn key_value(&self) -> &Arc<KeyValueDB> {
-			&self.key_value
-		}
-
-		fn blooms(&self) -> &blooms_db::Database {
-			&self.blooms
-		}
-
-		fn trace_blooms(&self) -> &blooms_db::Database {
-			&self.trace_blooms
-		}
-	}
-
 	let blooms_dir = TempDir::new("").unwrap();
 	let trace_blooms_dir = TempDir::new("").unwrap();
 
@@ -301,6 +301,26 @@ pub fn new_db() -> Arc<BlockChainDB> {
 	Arc::new(db)
 }
 
+/// Creates a new temporary `BlockChainDB` on FS
+pub fn new_temp_db(tempdir: &Path) -> Arc<BlockChainDB> {
+	let blooms_dir = TempDir::new("").unwrap();
+	let trace_blooms_dir = TempDir::new("").unwrap();
+	let key_value_dir = tempdir.join("key_value");
+
+	let db_config = DatabaseConfig::with_columns(::db::NUM_COLUMNS);
+	let key_value_db = Database::open(&db_config, key_value_dir.to_str().unwrap()).unwrap();
+
+	let db = TestBlockChainDB {
+		blooms: blooms_db::Database::open(blooms_dir.path()).unwrap(),
+		trace_blooms: blooms_db::Database::open(trace_blooms_dir.path()).unwrap(),
+		_blooms_dir: blooms_dir,
+		_trace_blooms_dir: trace_blooms_dir,
+		key_value: Arc::new(key_value_db)
+	};
+
+	Arc::new(db)
+}
+
 /// Creates new instance of KeyValueDBHandler
 pub fn restoration_db_handler(config: kvdb_rocksdb::DatabaseConfig) -> Box<BlockChainDBHandler> {
 	struct RestorationDBHandler {

ethcore/src/verification/queue/mod.rs

@@ -583,6 +583,13 @@ impl<K: Kind> VerificationQueue<K> {
 		result
 	}
 
+	/// Returns true if there is nothing currently in the queue.
+	/// TODO [ToDr] Optimize to avoid locking
+	pub fn is_empty(&self) -> bool {
+		let v = &self.verification;
+		v.unverified.lock().is_empty() && v.verifying.lock().is_empty() && v.verified.lock().is_empty()
+	}
+
 	/// Get queue status.
 	pub fn queue_info(&self) -> QueueInfo {
 		use std::mem::size_of;

ethcore/src/verification/verification.rs

@@ -147,7 +147,9 @@ pub fn verify_block_family<C: BlockInfo + CallContract>(header: &Header, parent:
 	verify_uncles(params.block, params.block_provider, engine)?;
 
 	for tx in &params.block.transactions {
-		engine.machine().verify_transaction(tx, header, params.client)?;
+		// transactions are verified against the parent header since the current
+		// state wasn't available when the tx was created
+		engine.machine().verify_transaction(tx, parent, params.client)?;
 	}
 
 	Ok(())
@@ -173,7 +175,7 @@ fn verify_uncles(block: &PreverifiedBlock, bc: &BlockProvider, engine: &EthEngin
 		for _ in 0..engine.maximum_uncle_age() {
 			match bc.block_details(&hash) {
 				Some(details) => {
-					excluded.insert(details.parent.clone());
+					excluded.insert(details.parent);
 					let b = bc.block(&hash)
 						.expect("parent already known to be stored; qed");
 					excluded.extend(b.uncle_hashes());
@@ -245,17 +247,17 @@ fn verify_uncles(block: &PreverifiedBlock, bc: &BlockProvider, engine: &EthEngin
 
 /// Phase 4 verification. Check block information against transaction enactment results,
 pub fn verify_block_final(expected: &Header, got: &Header) -> Result<(), Error> {
+	if expected.state_root() != got.state_root() {
+		return Err(From::from(BlockError::InvalidStateRoot(Mismatch { expected: *expected.state_root(), found: *got.state_root() })))
+	}
 	if expected.gas_used() != got.gas_used() {
-		return Err(From::from(BlockError::InvalidGasUsed(Mismatch { expected: expected.gas_used().clone(), found: got.gas_used().clone() })))
+		return Err(From::from(BlockError::InvalidGasUsed(Mismatch { expected: *expected.gas_used(), found: *got.gas_used() })))
 	}
 	if expected.log_bloom() != got.log_bloom() {
-		return Err(From::from(BlockError::InvalidLogBloom(Mismatch { expected: expected.log_bloom().clone(), found: got.log_bloom().clone() })))
-	}
-	if expected.state_root() != got.state_root() {
-		return Err(From::from(BlockError::InvalidStateRoot(Mismatch { expected: expected.state_root().clone(), found: got.state_root().clone() })))
+		return Err(From::from(BlockError::InvalidLogBloom(Mismatch { expected: *expected.log_bloom(), found: *got.log_bloom() })))
 	}
 	if expected.receipts_root() != got.receipts_root() {
-		return Err(From::from(BlockError::InvalidReceiptsRoot(Mismatch { expected: expected.receipts_root().clone(), found: got.receipts_root().clone() })))
+		return Err(From::from(BlockError::InvalidReceiptsRoot(Mismatch { expected: *expected.receipts_root(), found: *got.receipts_root() })))
 	}
 	Ok(())
 }
@@ -275,11 +277,16 @@ pub fn verify_header_params(header: &Header, engine: &EthEngine, is_full: bool,
 		return Err(From::from(BlockError::RidiculousNumber(OutOfBounds { max: Some(From::from(BlockNumber::max_value())), min: None, found: header.number() })))
 	}
 	if header.gas_used() > header.gas_limit() {
-		return Err(From::from(BlockError::TooMuchGasUsed(OutOfBounds { max: Some(header.gas_limit().clone()), min: None, found: header.gas_used().clone() })));
+		return Err(From::from(BlockError::TooMuchGasUsed(OutOfBounds { max: Some(*header.gas_limit()), min: None, found: *header.gas_used() })));
 	}
 	let min_gas_limit = engine.params().min_gas_limit;
 	if header.gas_limit() < &min_gas_limit {
-		return Err(From::from(BlockError::InvalidGasLimit(OutOfBounds { min: Some(min_gas_limit), max: None, found: header.gas_limit().clone() })));
+		return Err(From::from(BlockError::InvalidGasLimit(OutOfBounds { min: Some(min_gas_limit), max: None, found: *header.gas_limit() })));
+	}
+	if let Some(limit) = engine.maximum_gas_limit() {
+		if header.gas_limit() > &limit {
+			return Err(From::from(::error::BlockError::InvalidGasLimit(OutOfBounds { min: None, max: Some(limit), found: *header.gas_limit() })));
+		}
 	}
 	let maximum_extra_data_size = engine.maximum_extra_data_size();
 	if header.number() != 0 && header.extra_data().len() > maximum_extra_data_size {
@@ -336,7 +343,7 @@ fn verify_parent(header: &Header, parent: &Header, engine: &EthEngine) -> Result
 	let min_gas = parent_gas_limit - parent_gas_limit / gas_limit_divisor;
 	let max_gas = parent_gas_limit + parent_gas_limit / gas_limit_divisor;
 	if header.gas_limit() <= &min_gas || header.gas_limit() >= &max_gas {
-		return Err(From::from(BlockError::InvalidGasLimit(OutOfBounds { min: Some(min_gas), max: Some(max_gas), found: header.gas_limit().clone() })));
+		return Err(From::from(BlockError::InvalidGasLimit(OutOfBounds { min: Some(min_gas), max: Some(max_gas), found: *header.gas_limit() })));
 	}
 
 	Ok(())
@@ -520,7 +527,7 @@ mod tests {
 		// is fine.
 		let client = ::client::TestBlockChainClient::default();
 		let parent = bc.block_header_data(header.parent_hash())
-			.ok_or(BlockError::UnknownParent(header.parent_hash().clone()))?
+			.ok_or(BlockError::UnknownParent(*header.parent_hash()))?
 			.decode()?;
 
 		let block = PreverifiedBlock {

ethcore/sync/src/api.rs

@@ -14,7 +14,7 @@
 // You should have received a copy of the GNU General Public License
 // along with Parity.  If not, see <http://www.gnu.org/licenses/>.
 
-use std::sync::Arc;
+use std::sync::{Arc, mpsc, atomic};
 use std::collections::{HashMap, BTreeMap};
 use std::io;
 use std::ops::Range;
@@ -33,10 +33,10 @@ use ethcore::client::{BlockChainClient, ChainNotify, ChainRoute, ChainMessageTyp
 use ethcore::snapshot::SnapshotService;
 use ethcore::header::BlockNumber;
 use sync_io::NetSyncIo;
-use chain::{ChainSync, SyncStatus as EthSyncStatus};
+use chain::{ChainSyncApi, SyncStatus as EthSyncStatus};
 use std::net::{SocketAddr, AddrParseError};
 use std::str::FromStr;
-use parking_lot::RwLock;
+use parking_lot::{RwLock, Mutex};
 use chain::{ETH_PROTOCOL_VERSION_63, ETH_PROTOCOL_VERSION_62,
 	PAR_PROTOCOL_VERSION_1, PAR_PROTOCOL_VERSION_2, PAR_PROTOCOL_VERSION_3,
 	PRIVATE_TRANSACTION_PACKET, SIGNED_PRIVATE_TRANSACTION_PACKET};
@@ -228,6 +228,37 @@ impl AttachedProtocol {
 	}
 }
 
+/// A prioritized tasks run in a specialised timer.
+/// Every task should be completed within a hard deadline,
+/// if it's not it's either cancelled or split into multiple tasks.
+/// NOTE These tasks might not complete at all, so anything
+/// that happens here should work even if the task is cancelled.
+#[derive(Debug)]
+pub enum PriorityTask {
+	/// Propagate given block
+	PropagateBlock {
+		/// When the task was initiated
+		started: ::std::time::Instant,
+		/// Raw block RLP to propagate
+		block: Bytes,
+		/// Block hash
+		hash: H256,
+		/// Blocks difficulty
+		difficulty: U256,
+	},
+	/// Propagate a list of transactions
+	PropagateTransactions(::std::time::Instant, Arc<atomic::AtomicBool>),
+}
+impl PriorityTask {
+	/// Mark the task as being processed, right after it's retrieved from the queue.
+	pub fn starting(&self) {
+		match *self {
+			PriorityTask::PropagateTransactions(_, ref is_ready) => is_ready.store(true, atomic::Ordering::SeqCst),
+			_ => {},
+		}
+	}
+}
+
 /// EthSync initialization parameters.
 pub struct Params {
 	/// Configuration.
@@ -260,16 +291,16 @@ pub struct EthSync {
 	subprotocol_name: [u8; 3],
 	/// Light subprotocol name.
 	light_subprotocol_name: [u8; 3],
+	/// Priority tasks notification channel
+	priority_tasks: Mutex<mpsc::Sender<PriorityTask>>,
 }
 
 fn light_params(
 	network_id: u64,
-	max_peers: u32,
+	median_peers: f64,
 	pruning_info: PruningInfo,
 	sample_store: Option<Box<SampleStore>>,
 ) -> LightParams {
-	const MAX_LIGHTSERV_LOAD: f64 = 0.5;
-
 	let mut light_params = LightParams {
 		network_id: network_id,
 		config: Default::default(),
@@ -282,9 +313,7 @@ fn light_params(
 		sample_store: sample_store,
 	};
 
-	let max_peers = ::std::cmp::max(max_peers, 1);
-	light_params.config.load_share = MAX_LIGHTSERV_LOAD / max_peers as f64;
-
+	light_params.config.median_peers = median_peers;
 	light_params
 }
 
@@ -301,9 +330,10 @@ impl EthSync {
 					.map(|mut p| { p.push("request_timings"); light_net::FileStore(p) })
 					.map(|store| Box::new(store) as Box<_>);
 
+				let median_peers = (params.network_config.min_peers + params.network_config.max_peers) as f64 / 2.0;
 				let light_params = light_params(
 					params.config.network_id,
-					params.network_config.max_peers,
+					median_peers,
 					pruning_info,
 					sample_store,
 				);
@@ -315,13 +345,19 @@ impl EthSync {
 			})
 		};
 
-		let chain_sync = ChainSync::new(params.config, &*params.chain, params.private_tx_handler.clone());
+		let (priority_tasks_tx, priority_tasks_rx) = mpsc::channel();
+		let sync = ChainSyncApi::new(
+			params.config,
+			&*params.chain,
+			params.private_tx_handler.clone(),
+			priority_tasks_rx,
+		);
 		let service = NetworkService::new(params.network_config.clone().into_basic()?, connection_filter)?;
 
 		let sync = Arc::new(EthSync {
 			network: service,
 			eth_handler: Arc::new(SyncProtocolHandler {
-				sync: RwLock::new(chain_sync),
+				sync,
 				chain: params.chain,
 				snapshot_service: params.snapshot_service,
 				overlay: RwLock::new(HashMap::new()),
@@ -330,26 +366,32 @@ impl EthSync {
 			subprotocol_name: params.config.subprotocol_name,
 			light_subprotocol_name: params.config.light_subprotocol_name,
 			attached_protos: params.attached_protos,
+			priority_tasks: Mutex::new(priority_tasks_tx),
 		});
 
 		Ok(sync)
 	}
+
+	/// Priority tasks producer
+	pub fn priority_tasks(&self) -> mpsc::Sender<PriorityTask> {
+		self.priority_tasks.lock().clone()
+	}
 }
 
 impl SyncProvider for EthSync {
 	/// Get sync status
 	fn status(&self) -> EthSyncStatus {
-		self.eth_handler.sync.read().status()
+		self.eth_handler.sync.status()
 	}
 
 	/// Get sync peers
 	fn peers(&self) -> Vec<PeerInfo> {
 		self.network.with_context_eval(self.subprotocol_name, |ctx| {
 			let peer_ids = self.network.connected_peers();
-			let eth_sync = self.eth_handler.sync.read();
 			let light_proto = self.light_proto.as_ref();
 
-			peer_ids.into_iter().filter_map(|peer_id| {
+			let peer_info = self.eth_handler.sync.peer_info(&peer_ids);
+			peer_ids.into_iter().zip(peer_info).filter_map(|(peer_id, peer_info)| {
 				let session_info = match ctx.session_info(peer_id) {
 					None => return None,
 					Some(info) => info,
@@ -361,7 +403,7 @@ impl SyncProvider for EthSync {
 					capabilities: session_info.peer_capabilities.into_iter().map(|c| c.to_string()).collect(),
 					remote_address: session_info.remote_address,
 					local_address: session_info.local_address,
-					eth_info: eth_sync.peer_info(&peer_id),
+					eth_info: peer_info,
 					pip_info: light_proto.as_ref().and_then(|lp| lp.peer_status(peer_id)).map(Into::into),
 				})
 			}).collect()
@@ -373,17 +415,17 @@ impl SyncProvider for EthSync {
 	}
 
 	fn transactions_stats(&self) -> BTreeMap<H256, TransactionStats> {
-		let sync = self.eth_handler.sync.read();
-		sync.transactions_stats()
-			.iter()
-			.map(|(hash, stats)| (*hash, stats.into()))
-			.collect()
+		self.eth_handler.sync.transactions_stats()
 	}
 }
 
 const PEERS_TIMER: TimerToken = 0;
-const SYNC_TIMER: TimerToken = 1;
-const TX_TIMER: TimerToken = 2;
+const MAINTAIN_SYNC_TIMER: TimerToken = 1;
+const CONTINUE_SYNC_TIMER: TimerToken = 2;
+const TX_TIMER: TimerToken = 3;
+const PRIORITY_TIMER: TimerToken = 4;
+
+pub(crate) const PRIORITY_TIMER_INTERVAL: Duration = Duration::from_millis(250);
 
 struct SyncProtocolHandler {
 	/// Shared blockchain client.
@@ -391,7 +433,7 @@ struct SyncProtocolHandler {
 	/// Shared snapshot service.
 	snapshot_service: Arc<SnapshotService>,
 	/// Sync strategy
-	sync: RwLock<ChainSync>,
+	sync: ChainSyncApi,
 	/// Chain overlay used to cache data such as fork block.
 	overlay: RwLock<HashMap<BlockNumber, Bytes>>,
 }
@@ -400,13 +442,16 @@ impl NetworkProtocolHandler for SyncProtocolHandler {
 	fn initialize(&self, io: &NetworkContext) {
 		if io.subprotocol_name() != WARP_SYNC_PROTOCOL_ID {
 			io.register_timer(PEERS_TIMER, Duration::from_millis(700)).expect("Error registering peers timer");
-			io.register_timer(SYNC_TIMER, Duration::from_millis(1100)).expect("Error registering sync timer");
+			io.register_timer(MAINTAIN_SYNC_TIMER, Duration::from_millis(1100)).expect("Error registering sync timer");
+			io.register_timer(CONTINUE_SYNC_TIMER, Duration::from_millis(2500)).expect("Error registering sync timer");
 			io.register_timer(TX_TIMER, Duration::from_millis(1300)).expect("Error registering transactions timer");
+
+			io.register_timer(PRIORITY_TIMER, PRIORITY_TIMER_INTERVAL).expect("Error registering peers timer");
 		}
 	}
 
 	fn read(&self, io: &NetworkContext, peer: &PeerId, packet_id: u8, data: &[u8]) {
-		ChainSync::dispatch_packet(&self.sync, &mut NetSyncIo::new(io, &*self.chain, &*self.snapshot_service, &self.overlay), *peer, packet_id, data);
+		self.sync.dispatch_packet(&mut NetSyncIo::new(io, &*self.chain, &*self.snapshot_service, &self.overlay), *peer, packet_id, data);
 	}
 
 	fn connected(&self, io: &NetworkContext, peer: &PeerId) {
@@ -431,16 +476,28 @@ impl NetworkProtocolHandler for SyncProtocolHandler {
 		let mut io = NetSyncIo::new(io, &*self.chain, &*self.snapshot_service, &self.overlay);
 		match timer {
 			PEERS_TIMER => self.sync.write().maintain_peers(&mut io),
-			SYNC_TIMER => self.sync.write().maintain_sync(&mut io),
-			TX_TIMER => {
-				self.sync.write().propagate_new_transactions(&mut io);
-			},
+			MAINTAIN_SYNC_TIMER => self.sync.write().maintain_sync(&mut io),
+			CONTINUE_SYNC_TIMER => self.sync.write().continue_sync(&mut io),
+			TX_TIMER => self.sync.write().propagate_new_transactions(&mut io),
+			PRIORITY_TIMER => self.sync.process_priority_queue(&mut io),
 			_ => warn!("Unknown timer {} triggered.", timer),
 		}
 	}
 }
 
 impl ChainNotify for EthSync {
+	fn block_pre_import(&self, bytes: &Bytes, hash: &H256, difficulty: &U256) {
+		let task = PriorityTask::PropagateBlock {
+			started: ::std::time::Instant::now(),
+			block: bytes.clone(),
+			hash: *hash,
+			difficulty: *difficulty,
+		};
+		if let Err(e) = self.priority_tasks.lock().send(task) {
+			warn!(target: "sync", "Unexpected error during priority block propagation: {:?}", e);
+		}
+	}
+
 	fn new_blocks(&self,
 		imported: Vec<H256>,
 		invalid: Vec<H256>,
@@ -940,19 +997,3 @@ impl LightSyncProvider for LightSync {
 		Default::default() // TODO
 	}
 }
-
-#[cfg(test)]
-mod tests {
-	use super::*;
-
-	#[test]
-	fn light_params_load_share_depends_on_max_peers() {
-		let pruning_info = PruningInfo {
-			earliest_chain: 0,
-			earliest_state: 0,
-		};
-		let params1 = light_params(0, 10, pruning_info.clone(), None);
-		let params2 = light_params(0, 20, pruning_info, None);
-		assert!(params1.config.load_share > params2.config.load_share)
-	}
-}

ethcore/sync/src/chain/handler.rs

@@ -29,7 +29,6 @@ use rlp::Rlp;
 use snapshot::ChunkType;
 use std::cmp;
 use std::mem;
-use std::collections::HashSet;
 use std::time::Instant;
 use sync_io::SyncIo;
 
@@ -58,7 +57,6 @@ use super::{
 	SNAPSHOT_DATA_PACKET,
 	SNAPSHOT_MANIFEST_PACKET,
 	STATUS_PACKET,
-	TRANSACTIONS_PACKET,
 };
 
 /// The Chain Sync Handler: handles responses from peers
@@ -67,14 +65,9 @@ pub struct SyncHandler;
 impl SyncHandler {
 	/// Handle incoming packet from peer
 	pub fn on_packet(sync: &mut ChainSync, io: &mut SyncIo, peer: PeerId, packet_id: u8, data: &[u8]) {
-		if packet_id != STATUS_PACKET && !sync.peers.contains_key(&peer) {
-			debug!(target:"sync", "Unexpected packet {} from unregistered peer: {}:{}", packet_id, peer, io.peer_info(peer));
-			return;
-		}
 		let rlp = Rlp::new(data);
 		let result = match packet_id {
 			STATUS_PACKET => SyncHandler::on_peer_status(sync, io, peer, &rlp),
-			TRANSACTIONS_PACKET => SyncHandler::on_peer_transactions(sync, io, peer, &rlp),
 			BLOCK_HEADERS_PACKET => SyncHandler::on_peer_block_headers(sync, io, peer, &rlp),
 			BLOCK_BODIES_PACKET => SyncHandler::on_peer_block_bodies(sync, io, peer, &rlp),
 			RECEIPTS_PACKET => SyncHandler::on_peer_block_receipts(sync, io, peer, &rlp),
@@ -104,15 +97,12 @@ impl SyncHandler {
 				sync.sync_peer(io, peer, false);
 			},
 		}
-		// give tasks to other peers
-		sync.continue_sync(io);
 	}
 
 	/// Called when peer sends us new consensus packet
-	pub fn on_consensus_packet(io: &mut SyncIo, peer_id: PeerId, r: &Rlp) -> Result<(), PacketDecodeError> {
+	pub fn on_consensus_packet(io: &mut SyncIo, peer_id: PeerId, r: &Rlp) {
 		trace!(target: "sync", "Received consensus packet from {:?}", peer_id);
 		io.chain().queue_consensus_message(r.as_raw().to_vec());
-		Ok(())
 	}
 
 	/// Called by peer when it is disconnecting
@@ -578,8 +568,8 @@ impl SyncHandler {
 			asking_blocks: Vec::new(),
 			asking_hash: None,
 			ask_time: Instant::now(),
-			last_sent_transactions: HashSet::new(),
-			last_sent_private_transactions: HashSet::new(),
+			last_sent_transactions: Default::default(),
+			last_sent_private_transactions: Default::default(),
 			expired: false,
 			confirmation: if sync.fork_block.is_none() { ForkConfirmation::Confirmed } else { ForkConfirmation::Unconfirmed },
 			asking_snapshot_data: None,
@@ -635,7 +625,7 @@ impl SyncHandler {
 	}
 
 	/// Called when peer sends us new transactions
-	fn on_peer_transactions(sync: &mut ChainSync, io: &mut SyncIo, peer_id: PeerId, r: &Rlp) -> Result<(), DownloaderImportError> {
+	pub fn on_peer_transactions(sync: &ChainSync, io: &mut SyncIo, peer_id: PeerId, r: &Rlp) -> Result<(), PacketDecodeError> {
 		// Accept transactions only when fully synced
 		if !io.is_chain_queue_empty() || (sync.state != SyncState::Idle && sync.state != SyncState::NewBlocks) {
 			trace!(target: "sync", "{} Ignoring transactions while syncing", peer_id);

ethcore/sync/src/chain/mod.rs

@@ -92,17 +92,17 @@ mod propagator;
 mod requester;
 mod supplier;
 
-use std::sync::Arc;
-use std::collections::{HashSet, HashMap};
+use std::sync::{Arc, mpsc};
+use std::collections::{HashSet, HashMap, BTreeMap};
 use std::cmp;
 use std::time::{Duration, Instant};
 use hash::keccak;
 use heapsize::HeapSizeOf;
 use ethereum_types::{H256, U256};
-use fastmap::H256FastMap;
-use parking_lot::RwLock;
+use fastmap::{H256FastMap, H256FastSet};
+use parking_lot::{Mutex, RwLock, RwLockWriteGuard};
 use bytes::Bytes;
-use rlp::{Rlp, RlpStream, DecoderError};
+use rlp::{RlpStream, DecoderError};
 use network::{self, PeerId, PacketId};
 use ethcore::header::{BlockNumber};
 use ethcore::client::{BlockChainClient, BlockStatus, BlockId, BlockChainInfo, BlockQueueInfo};
@@ -112,7 +112,7 @@ use super::{WarpSync, SyncConfig};
 use block_sync::{BlockDownloader, DownloadAction};
 use rand::Rng;
 use snapshot::{Snapshot};
-use api::{EthProtocolInfo as PeerInfoDigest, WARP_SYNC_PROTOCOL_ID};
+use api::{EthProtocolInfo as PeerInfoDigest, WARP_SYNC_PROTOCOL_ID, PriorityTask};
 use private_tx::PrivateTxHandler;
 use transactions_stats::{TransactionsStats, Stats as TransactionStats};
 use transaction::UnverifiedTransaction;
@@ -120,7 +120,7 @@ use transaction::UnverifiedTransaction;
 use self::handler::SyncHandler;
 use self::propagator::SyncPropagator;
 use self::requester::SyncRequester;
-use self::supplier::SyncSupplier;
+pub(crate) use self::supplier::SyncSupplier;
 
 known_heap_size!(0, PeerInfo);
 
@@ -187,6 +187,11 @@ const FORK_HEADER_TIMEOUT: Duration = Duration::from_secs(3);
 const SNAPSHOT_MANIFEST_TIMEOUT: Duration = Duration::from_secs(5);
 const SNAPSHOT_DATA_TIMEOUT: Duration = Duration::from_secs(120);
 
+/// Defines how much time we have to complete priority transaction or block propagation.
+/// after the deadline is reached the task is considered finished
+/// (so we might sent only to some part of the peers we originally intended to send to)
+const PRIORITY_TASK_DEADLINE: Duration = Duration::from_millis(100);
+
 #[derive(Copy, Clone, Eq, PartialEq, Debug)]
 /// Sync state
 pub enum SyncState {
@@ -323,9 +328,9 @@ pub struct PeerInfo {
 	/// Request timestamp
 	ask_time: Instant,
 	/// Holds a set of transactions recently sent to this peer to avoid spamming.
-	last_sent_transactions: HashSet<H256>,
+	last_sent_transactions: H256FastSet,
 	/// Holds a set of private transactions and their signatures recently sent to this peer to avoid spamming.
-	last_sent_private_transactions: HashSet<H256>,
+	last_sent_private_transactions: H256FastSet,
 	/// Pending request is expired and result should be ignored
 	expired: bool,
 	/// Peer fork confirmation status
@@ -375,6 +380,217 @@ pub mod random {
 pub type RlpResponseResult = Result<Option<(PacketId, RlpStream)>, PacketDecodeError>;
 pub type Peers = HashMap<PeerId, PeerInfo>;
 
+/// Thread-safe wrapper for `ChainSync`.
+///
+/// NOTE always lock in order of fields declaration
+pub struct ChainSyncApi {
+	/// Priority tasks queue
+	priority_tasks: Mutex<mpsc::Receiver<PriorityTask>>,
+	/// The rest of sync data
+	sync: RwLock<ChainSync>,
+}
+
+impl ChainSyncApi {
+	/// Creates new `ChainSyncApi`
+	pub fn new(
+		config: SyncConfig,
+		chain: &BlockChainClient,
+		private_tx_handler: Arc<PrivateTxHandler>,
+		priority_tasks: mpsc::Receiver<PriorityTask>,
+	) -> Self {
+		ChainSyncApi {
+			sync: RwLock::new(ChainSync::new(config, chain, private_tx_handler)),
+			priority_tasks: Mutex::new(priority_tasks),
+		}
+	}
+
+	/// Gives `write` access to underlying `ChainSync`
+	pub fn write(&self) -> RwLockWriteGuard<ChainSync> {
+		self.sync.write()
+	}
+
+	/// Returns info about given list of peers
+	pub fn peer_info(&self, ids: &[PeerId]) -> Vec<Option<PeerInfoDigest>> {
+		let sync = self.sync.read();
+		ids.iter().map(|id| sync.peer_info(id)).collect()
+	}
+
+	/// Returns synchonization status
+	pub fn status(&self) -> SyncStatus {
+		self.sync.read().status()
+	}
+
+	/// Returns transactions propagation statistics
+	pub fn transactions_stats(&self) -> BTreeMap<H256, ::TransactionStats> {
+		self.sync.read().transactions_stats()
+			.iter()
+			.map(|(hash, stats)| (*hash, stats.into()))
+			.collect()
+	}
+
+	/// Dispatch incoming requests and responses
+	pub fn dispatch_packet(&self, io: &mut SyncIo, peer: PeerId, packet_id: u8, data: &[u8]) {
+		SyncSupplier::dispatch_packet(&self.sync, io, peer, packet_id, data)
+	}
+
+	/// Process a priority propagation queue.
+	/// This task is run from a timer and should be time constrained.
+	/// Hence we set up a deadline for the execution and cancel the task if the deadline is exceeded.
+	///
+	/// NOTE This method should only handle stuff that can be canceled and would reach other peers
+	/// by other means.
+	pub fn process_priority_queue(&self, io: &mut SyncIo) {
+		fn check_deadline(deadline: Instant) -> Option<Duration> {
+			let now = Instant::now();
+			if now > deadline {
+				None
+			} else {
+				Some(deadline - now)
+			}
+		}
+
+		// deadline to get the task from the queue
+		let deadline = Instant::now() + ::api::PRIORITY_TIMER_INTERVAL;
+		let mut work = || {
+			let task = {
+				let tasks = self.priority_tasks.try_lock_until(deadline)?;
+				let left = check_deadline(deadline)?;
+				tasks.recv_timeout(left).ok()?
+			};
+			task.starting();
+			// wait for the sync lock until deadline,
+			// note we might drop the task here if we won't manage to acquire the lock.
+			let mut sync = self.sync.try_write_until(deadline)?;
+			// since we already have everything let's use a different deadline
+			// to do the rest of the job now, so that previous work is not wasted.
+			let deadline = Instant::now() + PRIORITY_TASK_DEADLINE;
+			let as_ms = move |prev| {
+				let dur: Duration = Instant::now() - prev;
+				dur.as_secs() * 1_000 + dur.subsec_millis() as u64
+			};
+			match task {
+				// NOTE We can't simply use existing methods,
+				// cause the block is not in the DB yet.
+				PriorityTask::PropagateBlock { started, block, hash, difficulty } => {
+					// try to send to peers that are on the same block as us
+					// (they will most likely accept the new block).
+					let chain_info = io.chain().chain_info();
+					let total_difficulty = chain_info.total_difficulty + difficulty;
+					let rlp = ChainSync::create_block_rlp(&block, total_difficulty);
+					for peers in sync.get_peers(&chain_info, PeerState::SameBlock).chunks(10) {
+						check_deadline(deadline)?;
+						for peer in peers {
+							SyncPropagator::send_packet(io, *peer, NEW_BLOCK_PACKET, rlp.clone());
+							if let Some(ref mut peer) = sync.peers.get_mut(peer) {
+								peer.latest_hash = hash;
+							}
+						}
+					}
+					debug!(target: "sync", "Finished block propagation, took {}ms", as_ms(started));
+				},
+				PriorityTask::PropagateTransactions(time, _) => {
+					SyncPropagator::propagate_new_transactions(&mut sync, io, || {
+						check_deadline(deadline).is_some()
+					});
+					debug!(target: "sync", "Finished transaction propagation, took {}ms", as_ms(time));
+				},
+			}
+
+			Some(())
+		};
+
+		// Process as many items as we can until the deadline is reached.
+		loop {
+			if work().is_none() {
+				return;
+			}
+		}
+	}
+}
+
+// Static methods
+impl ChainSync {
+	/// creates rlp to send for the tree defined by 'from' and 'to' hashes
+	fn create_new_hashes_rlp(chain: &BlockChainClient, from: &H256, to: &H256) -> Option<Bytes> {
+		match chain.tree_route(from, to) {
+			Some(route) => {
+				let uncles = chain.find_uncles(from).unwrap_or_else(Vec::new);
+				match route.blocks.len() {
+					0 => None,
+					_ => {
+						let mut blocks = route.blocks;
+						blocks.extend(uncles);
+						let mut rlp_stream = RlpStream::new_list(blocks.len());
+						for block_hash in  blocks {
+							let mut hash_rlp = RlpStream::new_list(2);
+							let number = chain.block_header(BlockId::Hash(block_hash.clone()))
+								.expect("chain.tree_route and chain.find_uncles only return hahses of blocks that are in the blockchain. qed.").number();
+							hash_rlp.append(&block_hash);
+							hash_rlp.append(&number);
+							rlp_stream.append_raw(hash_rlp.as_raw(), 1);
+						}
+						Some(rlp_stream.out())
+					}
+				}
+			},
+			None => None
+		}
+	}
+
+	/// creates rlp from block bytes and total difficulty
+	fn create_block_rlp(bytes: &Bytes, total_difficulty: U256) -> Bytes {
+		let mut rlp_stream = RlpStream::new_list(2);
+		rlp_stream.append_raw(bytes, 1);
+		rlp_stream.append(&total_difficulty);
+		rlp_stream.out()
+	}
+
+	/// creates latest block rlp for the given client
+	fn create_latest_block_rlp(chain: &BlockChainClient) -> Bytes {
+		Self::create_block_rlp(
+			&chain.block(BlockId::Hash(chain.chain_info().best_block_hash))
+				.expect("Best block always exists").into_inner(),
+			chain.chain_info().total_difficulty
+		)
+	}
+
+	/// creates given hash block rlp for the given client
+	fn create_new_block_rlp(chain: &BlockChainClient, hash: &H256) -> Bytes {
+		Self::create_block_rlp(
+			&chain.block(BlockId::Hash(hash.clone())).expect("Block has just been sealed; qed").into_inner(),
+			chain.block_total_difficulty(BlockId::Hash(hash.clone())).expect("Block has just been sealed; qed.")
+		)
+	}
+
+	fn select_random_peers(peers: &[PeerId]) -> Vec<PeerId> {
+		// take sqrt(x) peers
+		let mut peers = peers.to_vec();
+		let mut count = (peers.len() as f64).powf(0.5).round() as usize;
+		count = cmp::min(count, MAX_PEERS_PROPAGATION);
+		count = cmp::max(count, MIN_PEERS_PROPAGATION);
+		random::new().shuffle(&mut peers);
+		peers.truncate(count);
+		peers
+	}
+
+	fn get_init_state(warp_sync: WarpSync, chain: &BlockChainClient) -> SyncState {
+		let best_block = chain.chain_info().best_block_number;
+		match warp_sync {
+			WarpSync::Enabled => SyncState::WaitingPeers,
+			WarpSync::OnlyAndAfter(block) if block > best_block => SyncState::WaitingPeers,
+			_ => SyncState::Idle,
+		}
+	}
+}
+
+/// A peer query method for getting a list of peers
+enum PeerState {
+	/// Peer is on different hash than us
+	Lagging,
+	/// Peer is on the same block as us
+	SameBlock
+}
+
 /// Blockchain sync handler.
 /// See module documentation for more details.
 pub struct ChainSync {
@@ -417,10 +633,14 @@ pub struct ChainSync {
 
 impl ChainSync {
 	/// Create a new instance of syncing strategy.
-	pub fn new(config: SyncConfig, chain: &BlockChainClient, private_tx_handler: Arc<PrivateTxHandler>) -> ChainSync {
+	pub fn new(
+		config: SyncConfig,
+		chain: &BlockChainClient,
+		private_tx_handler: Arc<PrivateTxHandler>,
+	) -> Self {
 		let chain_info = chain.chain_info();
 		let best_block = chain.chain_info().best_block_number;
-		let state = ChainSync::get_init_state(config.warp_sync, chain);
+		let state = Self::get_init_state(config.warp_sync, chain);
 
 		let mut sync = ChainSync {
 			state,
@@ -445,15 +665,6 @@ impl ChainSync {
 		sync
 	}
 
-	fn get_init_state(warp_sync: WarpSync, chain: &BlockChainClient) -> SyncState {
-		let best_block = chain.chain_info().best_block_number;
-		match warp_sync {
-			WarpSync::Enabled => SyncState::WaitingPeers,
-			WarpSync::OnlyAndAfter(block) if block > best_block => SyncState::WaitingPeers,
-			_ => SyncState::Idle,
-		}
-	}
-
 	/// Returns synchonization status
 	pub fn status(&self) -> SyncStatus {
 		let last_imported_number = self.new_blocks.last_imported_block_number();
@@ -521,7 +732,7 @@ impl ChainSync {
 				}
 			}
 		}
-		self.state = state.unwrap_or_else(|| ChainSync::get_init_state(self.warp_sync, io.chain()));
+		self.state = state.unwrap_or_else(|| Self::get_init_state(self.warp_sync, io.chain()));
 		// Reactivate peers only if some progress has been made
 		// since the last sync round of if starting fresh.
 		self.active_peers = self.peers.keys().cloned().collect();
@@ -655,37 +866,35 @@ impl ChainSync {
 	}
 
 	/// Resume downloading
-	fn continue_sync(&mut self, io: &mut SyncIo) {
-		// Collect active peers that can sync
-		let confirmed_peers: Vec<(PeerId, u8)> = self.peers.iter().filter_map(|(peer_id, peer)|
-			if peer.can_sync() {
-				Some((*peer_id, peer.protocol_version))
-			} else {
-				None
-			}
-		).collect();
-
-		trace!(
-			target: "sync",
-			"Syncing with peers: {} active, {} confirmed, {} total",
-			self.active_peers.len(), confirmed_peers.len(), self.peers.len()
-		);
-
+	pub fn continue_sync(&mut self, io: &mut SyncIo) {
 		if self.state == SyncState::Waiting {
 			trace!(target: "sync", "Waiting for the block queue");
 		} else if self.state == SyncState::SnapshotWaiting {
 			trace!(target: "sync", "Waiting for the snapshot restoration");
 		} else {
-			let mut peers: Vec<(PeerId, u8)> = confirmed_peers.iter().filter(|&&(peer_id, _)|
-				self.active_peers.contains(&peer_id)
-			).map(|v| *v).collect();
+			// Collect active peers that can sync
+			let mut peers: Vec<(PeerId, u8)> = self.peers.iter().filter_map(|(peer_id, peer)|
+				if peer.can_sync() && peer.asking == PeerAsking::Nothing && self.active_peers.contains(&peer_id) {
+					Some((*peer_id, peer.protocol_version))
+				} else {
+					None
+				}
+			).collect();
 
-			random::new().shuffle(&mut peers); //TODO: sort by rating
-			// prefer peers with higher protocol version
-			peers.sort_by(|&(_, ref v1), &(_, ref v2)| v1.cmp(v2));
+			if peers.len() > 0 {
+				trace!(
+					target: "sync",
+					"Syncing with peers: {} active, {} available, {} total",
+					self.active_peers.len(), peers.len(), self.peers.len()
+				);
 
-			for (peer_id, _) in peers {
-				self.sync_peer(io, peer_id, false);
+				random::new().shuffle(&mut peers); // TODO (#646): sort by rating
+				// prefer peers with higher protocol version
+				peers.sort_by(|&(_, ref v1), &(_, ref v2)| v1.cmp(v2));
+
+				for (peer_id, _) in peers {
+					self.sync_peer(io, peer_id, false);
+				}
 			}
 		}
 
@@ -970,6 +1179,12 @@ impl ChainSync {
 				self.state = SyncState::Blocks;
 				self.continue_sync(io);
 			},
+			SyncState::SnapshotData => match io.snapshot_service().status() {
+				RestorationStatus::Inactive | RestorationStatus::Failed => {
+					self.state = SyncState::SnapshotWaiting;
+				},
+				RestorationStatus::Initializing { .. } | RestorationStatus::Ongoing { .. } => (),
+			},
 			SyncState::SnapshotWaiting => {
 				match io.snapshot_service().status() {
 					RestorationStatus::Inactive => {
@@ -998,67 +1213,24 @@ impl ChainSync {
 		}
 	}
 
-	/// creates rlp to send for the tree defined by 'from' and 'to' hashes
-	fn create_new_hashes_rlp(chain: &BlockChainClient, from: &H256, to: &H256) -> Option<Bytes> {
-		match chain.tree_route(from, to) {
-			Some(route) => {
-				let uncles = chain.find_uncles(from).unwrap_or_else(Vec::new);
-				match route.blocks.len() {
-					0 => None,
-					_ => {
-						let mut blocks = route.blocks;
-						blocks.extend(uncles);
-						let mut rlp_stream = RlpStream::new_list(blocks.len());
-						for block_hash in  blocks {
-							let mut hash_rlp = RlpStream::new_list(2);
-							let number = chain.block_header(BlockId::Hash(block_hash.clone()))
-								.expect("chain.tree_route and chain.find_uncles only return hahses of blocks that are in the blockchain. qed.").number();
-							hash_rlp.append(&block_hash);
-							hash_rlp.append(&number);
-							rlp_stream.append_raw(hash_rlp.as_raw(), 1);
-						}
-						Some(rlp_stream.out())
-					}
-				}
-			},
-			None => None
-		}
+	/// returns peer ids that have different block than our chain
+	fn get_lagging_peers(&self, chain_info: &BlockChainInfo) -> Vec<PeerId> {
+		self.get_peers(chain_info, PeerState::Lagging)
 	}
 
-	/// creates rlp from block bytes and total difficulty
-	fn create_block_rlp(bytes: &Bytes, total_difficulty: U256) -> Bytes {
-		let mut rlp_stream = RlpStream::new_list(2);
-		rlp_stream.append_raw(bytes, 1);
-		rlp_stream.append(&total_difficulty);
-		rlp_stream.out()
-	}
-
-	/// creates latest block rlp for the given client
-	fn create_latest_block_rlp(chain: &BlockChainClient) -> Bytes {
-		ChainSync::create_block_rlp(
-			&chain.block(BlockId::Hash(chain.chain_info().best_block_hash))
-				.expect("Best block always exists").into_inner(),
-			chain.chain_info().total_difficulty
-		)
-	}
-
-	/// creates given hash block rlp for the given client
-	fn create_new_block_rlp(chain: &BlockChainClient, hash: &H256) -> Bytes {
-		ChainSync::create_block_rlp(
-			&chain.block(BlockId::Hash(hash.clone())).expect("Block has just been sealed; qed").into_inner(),
-			chain.block_total_difficulty(BlockId::Hash(hash.clone())).expect("Block has just been sealed; qed.")
-		)
-	}
-
-	/// returns peer ids that have different blocks than our chain
-	fn get_lagging_peers(&mut self, chain_info: &BlockChainInfo) -> Vec<PeerId> {
+	/// returns peer ids that have different or the same blocks than our chain
+	fn get_peers(&self, chain_info: &BlockChainInfo, peers: PeerState) -> Vec<PeerId> {
 		let latest_hash = chain_info.best_block_hash;
 		self
 			.peers
-			.iter_mut()
+			.iter()
 			.filter_map(|(&id, ref mut peer_info)| {
 				trace!(target: "sync", "Checking peer our best {} their best {}", latest_hash, peer_info.latest_hash);
-				if peer_info.latest_hash != latest_hash {
+				let matches = match peers {
+					PeerState::Lagging => peer_info.latest_hash != latest_hash,
+					PeerState::SameBlock => peer_info.latest_hash == latest_hash,
+				};
+				if matches {
 					Some(id)
 				} else {
 					None
@@ -1067,17 +1239,6 @@ impl ChainSync {
 			.collect::<Vec<_>>()
 	}
 
-	fn select_random_peers(peers: &[PeerId]) -> Vec<PeerId> {
-		// take sqrt(x) peers
-		let mut peers = peers.to_vec();
-		let mut count = (peers.len() as f64).powf(0.5).round() as usize;
-		count = cmp::min(count, MAX_PEERS_PROPAGATION);
-		count = cmp::max(count, MIN_PEERS_PROPAGATION);
-		random::new().shuffle(&mut peers);
-		peers.truncate(count);
-		peers
-	}
-
 	fn get_consensus_peers(&self) -> Vec<PeerId> {
 		self.peers.iter().filter_map(|(id, p)| if p.protocol_version >= PAR_PROTOCOL_VERSION_2.0 { Some(*id) } else { None }).collect()
 	}
@@ -1126,21 +1287,10 @@ impl ChainSync {
 		}
 	}
 
-	/// Dispatch incoming requests and responses
-	pub fn dispatch_packet(sync: &RwLock<ChainSync>, io: &mut SyncIo, peer: PeerId, packet_id: u8, data: &[u8]) {
-		SyncSupplier::dispatch_packet(sync, io, peer, packet_id, data)
-	}
-
 	pub fn on_packet(&mut self, io: &mut SyncIo, peer: PeerId, packet_id: u8, data: &[u8]) {
-		debug!(target: "sync", "{} -> Dispatching packet: {}", peer, packet_id);
 		SyncHandler::on_packet(self, io, peer, packet_id, data);
 	}
 
-	/// Called when peer sends us new consensus packet
-	pub fn on_consensus_packet(io: &mut SyncIo, peer_id: PeerId, r: &Rlp) -> Result<(), PacketDecodeError> {
-		SyncHandler::on_consensus_packet(io, peer_id, r)
-	}
-
 	/// Called by peer when it is disconnecting
 	pub fn on_peer_aborting(&mut self, io: &mut SyncIo, peer: PeerId) {
 		SyncHandler::on_peer_aborting(self, io, peer);
@@ -1152,8 +1302,16 @@ impl ChainSync {
 	}
 
 	/// propagates new transactions to all peers
-	pub fn propagate_new_transactions(&mut self, io: &mut SyncIo) -> usize {
-		SyncPropagator::propagate_new_transactions(self, io)
+	pub fn propagate_new_transactions(&mut self, io: &mut SyncIo) {
+		let deadline = Instant::now() + Duration::from_millis(500);
+		SyncPropagator::propagate_new_transactions(self, io, || {
+			if deadline > Instant::now() {
+				true
+			} else {
+				debug!(target: "sync", "Wasn't able to finish transaction propagation within a deadline.");
+				false
+			}
+		});
 	}
 
 	/// Broadcast consensus message to peers.
@@ -1169,7 +1327,7 @@ impl ChainSync {
 
 #[cfg(test)]
 pub mod tests {
-	use std::collections::{HashSet, VecDeque};
+	use std::collections::{VecDeque};
 	use ethkey;
 	use network::PeerId;
 	use tests::helpers::{TestIo};
@@ -1285,8 +1443,8 @@ pub mod tests {
 				asking_blocks: Vec::new(),
 				asking_hash: None,
 				ask_time: Instant::now(),
-				last_sent_transactions: HashSet::new(),
-				last_sent_private_transactions: HashSet::new(),
+				last_sent_transactions: Default::default(),
+				last_sent_private_transactions: Default::default(),
 				expired: false,
 				confirmation: super::ForkConfirmation::Confirmed,
 				snapshot_number: None,
@@ -1301,7 +1459,7 @@ pub mod tests {
 	fn finds_lagging_peers() {
 		let mut client = TestBlockChainClient::new();
 		client.add_blocks(100, EachBlockWith::Uncle);
-		let mut sync = dummy_sync_with_peer(client.block_hash_delta_minus(10), &client);
+		let sync = dummy_sync_with_peer(client.block_hash_delta_minus(10), &client);
 		let chain_info = client.chain_info();
 
 		let lagging_peers = sync.get_lagging_peers(&chain_info);
@@ -1441,3 +1599,4 @@ pub mod tests {
 		assert_eq!(status.status.transaction_count, 0);
 	}
 }
+

ethcore/sync/src/chain/propagator.rs

@@ -18,6 +18,7 @@ use bytes::Bytes;
 use ethereum_types::H256;
 use ethcore::client::BlockChainInfo;
 use ethcore::header::BlockNumber;
+use fastmap::H256FastSet;
 use network::{PeerId, PacketId};
 use rand::Rng;
 use rlp::{Encodable, RlpStream};
@@ -69,49 +70,51 @@ impl SyncPropagator {
 	/// propagates latest block to a set of peers
 	pub fn propagate_blocks(sync: &mut ChainSync, chain_info: &BlockChainInfo, io: &mut SyncIo, blocks: &[H256], peers: &[PeerId]) -> usize {
 		trace!(target: "sync", "Sending NewBlocks to {:?}", peers);
-		let mut sent = 0;
-		for peer_id in peers {
-			if blocks.is_empty() {
-				let rlp =  ChainSync::create_latest_block_rlp(io.chain());
-				SyncPropagator::send_packet(io, *peer_id, NEW_BLOCK_PACKET, rlp);
-			} else {
-				for h in blocks {
-					let rlp =  ChainSync::create_new_block_rlp(io.chain(), h);
-					SyncPropagator::send_packet(io, *peer_id, NEW_BLOCK_PACKET, rlp);
+		let sent = peers.len();
+		let mut send_packet = |io: &mut SyncIo, rlp: Bytes| {
+			for peer_id in peers {
+				SyncPropagator::send_packet(io, *peer_id, NEW_BLOCK_PACKET, rlp.clone());
+				if let Some(ref mut peer) = sync.peers.get_mut(peer_id) {
+					peer.latest_hash = chain_info.best_block_hash.clone();
 				}
 			}
-			if let Some(ref mut peer) = sync.peers.get_mut(peer_id) {
-				peer.latest_hash = chain_info.best_block_hash.clone();
+		};
+
+		if blocks.is_empty() {
+			let rlp =  ChainSync::create_latest_block_rlp(io.chain());
+			send_packet(io, rlp);
+		} else {
+			for h in blocks {
+				let rlp =  ChainSync::create_new_block_rlp(io.chain(), h);
+				send_packet(io, rlp);
 			}
-			sent += 1;
 		}
+
 		sent
 	}
 
 	/// propagates new known hashes to all peers
 	pub fn propagate_new_hashes(sync: &mut ChainSync, chain_info: &BlockChainInfo, io: &mut SyncIo, peers: &[PeerId]) -> usize {
 		trace!(target: "sync", "Sending NewHashes to {:?}", peers);
-		let mut sent = 0;
 		let last_parent = *io.chain().best_block_header().parent_hash();
+		let best_block_hash = chain_info.best_block_hash;
+		let rlp = match ChainSync::create_new_hashes_rlp(io.chain(), &last_parent, &best_block_hash) {
+			Some(rlp) => rlp,
+			None => return 0
+		};
+
+		let sent = peers.len();
 		for peer_id in peers {
-			sent += match ChainSync::create_new_hashes_rlp(io.chain(), &last_parent, &chain_info.best_block_hash) {
-				Some(rlp) => {
-					{
-						if let Some(ref mut peer) = sync.peers.get_mut(peer_id) {
-							peer.latest_hash = chain_info.best_block_hash.clone();
-						}
-					}
-					SyncPropagator::send_packet(io, *peer_id, NEW_BLOCK_HASHES_PACKET, rlp);
-					1
-				},
-				None => 0
+			if let Some(ref mut peer) = sync.peers.get_mut(peer_id) {
+				peer.latest_hash = best_block_hash;
 			}
+			SyncPropagator::send_packet(io, *peer_id, NEW_BLOCK_HASHES_PACKET, rlp.clone());
 		}
 		sent
 	}
 
 	/// propagates new transactions to all peers
-	pub fn propagate_new_transactions(sync: &mut ChainSync, io: &mut SyncIo) -> usize {
+	pub fn propagate_new_transactions<F: FnMut() -> bool>(sync: &mut ChainSync, io: &mut SyncIo, mut should_continue: F) -> usize {
 		// Early out if nobody to send to.
 		if sync.peers.is_empty() {
 			return 0;
@@ -122,6 +125,10 @@ impl SyncPropagator {
 			return 0;
 		}
 
+		if !should_continue() {
+			return 0;
+		}
+
 		let (transactions, service_transactions): (Vec<_>, Vec<_>) = transactions.iter()
 			.map(|tx| tx.signed())
 			.partition(|tx| !tx.gas_price.is_zero());
@@ -130,24 +137,34 @@ impl SyncPropagator {
 		let mut affected_peers = HashSet::new();
 		if !transactions.is_empty() {
 			let peers = SyncPropagator::select_peers_for_transactions(sync, |_| true);
-			affected_peers = SyncPropagator::propagate_transactions_to_peers(sync, io, peers, transactions);
+			affected_peers = SyncPropagator::propagate_transactions_to_peers(
+				sync, io, peers, transactions, &mut should_continue,
+			);
 		}
 
 		// most of times service_transactions will be empty
 		// => there's no need to merge packets
 		if !service_transactions.is_empty() {
 			let service_transactions_peers = SyncPropagator::select_peers_for_transactions(sync, |peer_id| accepts_service_transaction(&io.peer_info(*peer_id)));
-			let service_transactions_affected_peers = SyncPropagator::propagate_transactions_to_peers(sync, io, service_transactions_peers, service_transactions);
+			let service_transactions_affected_peers = SyncPropagator::propagate_transactions_to_peers(
+				sync, io, service_transactions_peers, service_transactions, &mut should_continue
+			);
 			affected_peers.extend(&service_transactions_affected_peers);
 		}
 
 		affected_peers.len()
 	}
 
-	fn propagate_transactions_to_peers(sync: &mut ChainSync, io: &mut SyncIo, peers: Vec<PeerId>, transactions: Vec<&SignedTransaction>) -> HashSet<PeerId> {
+	fn propagate_transactions_to_peers<F: FnMut() -> bool>(
+		sync: &mut ChainSync,
+		io: &mut SyncIo,
+		peers: Vec<PeerId>,
+		transactions: Vec<&SignedTransaction>,
+		mut should_continue: F,
+	) -> HashSet<PeerId> {
 		let all_transactions_hashes = transactions.iter()
 			.map(|tx| tx.hash())
-			.collect::<HashSet<H256>>();
+			.collect::<H256FastSet>();
 		let all_transactions_rlp = {
 			let mut packet = RlpStream::new_list(transactions.len());
 			for tx in &transactions { packet.append(&**tx); }
@@ -157,102 +174,104 @@ impl SyncPropagator {
 		// Clear old transactions from stats
 		sync.transactions_stats.retain(&all_transactions_hashes);
 
-		// sqrt(x)/x scaled to max u32
-		let block_number = io.chain().chain_info().best_block_number;
-
-		let lucky_peers = {
-			peers.into_iter()
-				.filter_map(|peer_id| {
-					let stats = &mut sync.transactions_stats;
-					let peer_info = sync.peers.get_mut(&peer_id)
-						.expect("peer_id is form peers; peers is result of select_peers_for_transactions; select_peers_for_transactions selects peers from self.peers; qed");
-
-					// Send all transactions
-					if peer_info.last_sent_transactions.is_empty() {
-						// update stats
-						for hash in &all_transactions_hashes {
-							let id = io.peer_session_info(peer_id).and_then(|info| info.id);
-							stats.propagated(hash, id, block_number);
-						}
-						peer_info.last_sent_transactions = all_transactions_hashes.clone();
-						return Some((peer_id, all_transactions_hashes.len(), all_transactions_rlp.clone()));
-					}
-
-					// Get hashes of all transactions to send to this peer
-					let to_send = all_transactions_hashes.difference(&peer_info.last_sent_transactions)
-						.cloned()
-						.collect::<HashSet<_>>();
-					if to_send.is_empty() {
-						return None;
-					}
-
-					// Construct RLP
-					let (packet, to_send) = {
-						let mut to_send = to_send;
-						let mut packet = RlpStream::new();
-						packet.begin_unbounded_list();
-						let mut pushed = 0;
-						for tx in &transactions {
-							let hash = tx.hash();
-							if to_send.contains(&hash) {
-								let mut transaction = RlpStream::new();
-								tx.rlp_append(&mut transaction);
-								let appended = packet.append_raw_checked(&transaction.drain(), 1, MAX_TRANSACTION_PACKET_SIZE);
-								if !appended {
-									// Maximal packet size reached just proceed with sending
-									debug!(target: "sync", "Transaction packet size limit reached. Sending incomplete set of {}/{} transactions.", pushed, to_send.len());
-									to_send = to_send.into_iter().take(pushed).collect();
-									break;
-								}
-								pushed += 1;
-							}
-						}
-						packet.complete_unbounded_list();
-						(packet, to_send)
-					};
-
-					// Update stats
-					let id = io.peer_session_info(peer_id).and_then(|info| info.id);
-					for hash in &to_send {
-						// update stats
-						stats.propagated(hash, id, block_number);
-					}
-
-					peer_info.last_sent_transactions = all_transactions_hashes
-						.intersection(&peer_info.last_sent_transactions)
-						.chain(&to_send)
-						.cloned()
-						.collect();
-					Some((peer_id, to_send.len(), packet.out()))
-				})
-				.collect::<Vec<_>>()
+		let send_packet = |io: &mut SyncIo, peer_id: PeerId, sent: usize, rlp: Bytes| {
+			let size = rlp.len();
+			SyncPropagator::send_packet(io, peer_id, TRANSACTIONS_PACKET, rlp);
+			trace!(target: "sync", "{:02} <- Transactions ({} entries; {} bytes)", peer_id, sent, size);
 		};
 
-		// Send RLPs
-		let mut peers = HashSet::new();
-		if lucky_peers.len() > 0 {
-			let mut max_sent = 0;
-			let lucky_peers_len = lucky_peers.len();
-			for (peer_id, sent, rlp) in lucky_peers {
-				peers.insert(peer_id);
-				let size = rlp.len();
-				SyncPropagator::send_packet(io, peer_id, TRANSACTIONS_PACKET, rlp);
-				trace!(target: "sync", "{:02} <- Transactions ({} entries; {} bytes)", peer_id, sent, size);
-				max_sent = cmp::max(max_sent, sent);
+		let block_number = io.chain().chain_info().best_block_number;
+		let mut sent_to_peers = HashSet::new();
+		let mut max_sent = 0;
+
+		// for every peer construct and send transactions packet
+		for peer_id in peers {
+			if !should_continue() {
+				debug!(target: "sync", "Sent up to {} transactions to {} peers.", max_sent, sent_to_peers.len());
+				return sent_to_peers;
 			}
-			debug!(target: "sync", "Sent up to {} transactions to {} peers.", max_sent, lucky_peers_len);
+
+			let stats = &mut sync.transactions_stats;
+			let peer_info = sync.peers.get_mut(&peer_id)
+				.expect("peer_id is form peers; peers is result of select_peers_for_transactions; select_peers_for_transactions selects peers from self.peers; qed");
+
+			// Send all transactions, if the peer doesn't know about anything
+			if peer_info.last_sent_transactions.is_empty() {
+				// update stats
+				for hash in &all_transactions_hashes {
+					let id = io.peer_session_info(peer_id).and_then(|info| info.id);
+					stats.propagated(hash, id, block_number);
+				}
+				peer_info.last_sent_transactions = all_transactions_hashes.clone();
+
+				send_packet(io, peer_id, all_transactions_hashes.len(), all_transactions_rlp.clone());
+				sent_to_peers.insert(peer_id);
+				max_sent = cmp::max(max_sent, all_transactions_hashes.len());
+				continue;
+			}
+
+			// Get hashes of all transactions to send to this peer
+			let to_send = all_transactions_hashes.difference(&peer_info.last_sent_transactions)
+				.cloned()
+				.collect::<HashSet<_>>();
+			if to_send.is_empty() {
+				continue;
+			}
+
+			// Construct RLP
+			let (packet, to_send) = {
+				let mut to_send = to_send;
+				let mut packet = RlpStream::new();
+				packet.begin_unbounded_list();
+				let mut pushed = 0;
+				for tx in &transactions {
+					let hash = tx.hash();
+					if to_send.contains(&hash) {
+						let mut transaction = RlpStream::new();
+						tx.rlp_append(&mut transaction);
+						let appended = packet.append_raw_checked(&transaction.drain(), 1, MAX_TRANSACTION_PACKET_SIZE);
+						if !appended {
+							// Maximal packet size reached just proceed with sending
+							debug!(target: "sync", "Transaction packet size limit reached. Sending incomplete set of {}/{} transactions.", pushed, to_send.len());
+							to_send = to_send.into_iter().take(pushed).collect();
+							break;
+						}
+						pushed += 1;
+					}
+				}
+				packet.complete_unbounded_list();
+				(packet, to_send)
+			};
+
+			// Update stats
+			let id = io.peer_session_info(peer_id).and_then(|info| info.id);
+			for hash in &to_send {
+				// update stats
+				stats.propagated(hash, id, block_number);
+			}
+
+			peer_info.last_sent_transactions = all_transactions_hashes
+				.intersection(&peer_info.last_sent_transactions)
+				.chain(&to_send)
+				.cloned()
+				.collect();
+			send_packet(io, peer_id, to_send.len(), packet.out());
+			sent_to_peers.insert(peer_id);
+			max_sent = cmp::max(max_sent, to_send.len());
+
 		}
 
-		peers
+		debug!(target: "sync", "Sent up to {} transactions to {} peers.", max_sent, sent_to_peers.len());
+		sent_to_peers
 	}
 
 	pub fn propagate_latest_blocks(sync: &mut ChainSync, io: &mut SyncIo, sealed: &[H256]) {
 		let chain_info = io.chain().chain_info();
 		if (((chain_info.best_block_number as i64) - (sync.last_sent_block_number as i64)).abs() as BlockNumber) < MAX_PEER_LAG_PROPAGATION {
-			let mut peers = sync.get_lagging_peers(&chain_info);
+			let peers = sync.get_lagging_peers(&chain_info);
 			if sealed.is_empty() {
 				let hashes = SyncPropagator::propagate_new_hashes(sync, &chain_info, io, &peers);
-				peers = ChainSync::select_random_peers(&peers);
+				let peers = ChainSync::select_random_peers(&peers);
 				let blocks = SyncPropagator::propagate_blocks(sync, &chain_info, io, sealed, &peers);
 				if blocks != 0 || hashes != 0 {
 					trace!(target: "sync", "Sent latest {} blocks and {} hashes to peers.", blocks, hashes);
@@ -318,7 +337,7 @@ impl SyncPropagator {
 	}
 
 	/// Generic packet sender
-	fn send_packet(sync: &mut SyncIo, peer_id: PeerId, packet_id: PacketId, packet: Bytes) {
+	pub fn send_packet(sync: &mut SyncIo, peer_id: PeerId, packet_id: PacketId, packet: Bytes) {
 		if let Err(e) = sync.send(peer_id, packet_id, packet) {
 			debug!(target:"sync", "Error sending packet: {:?}", e);
 			sync.disconnect_peer(peer_id);
@@ -419,8 +438,8 @@ mod tests {
 				asking_blocks: Vec::new(),
 				asking_hash: None,
 				ask_time: Instant::now(),
-				last_sent_transactions: HashSet::new(),
-				last_sent_private_transactions: HashSet::new(),
+				last_sent_transactions: Default::default(),
+				last_sent_private_transactions: Default::default(),
 				expired: false,
 				confirmation: ForkConfirmation::Confirmed,
 				snapshot_number: None,
@@ -447,13 +466,13 @@ mod tests {
 		let queue = RwLock::new(VecDeque::new());
 		let ss = TestSnapshotService::new();
 		let mut io = TestIo::new(&mut client, &ss, &queue, None);
-		let peer_count = SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+		let peer_count = SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 		// Try to propagate same transactions for the second time
-		let peer_count2 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+		let peer_count2 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 		// Even after new block transactions should not be propagated twice
 		sync.chain_new_blocks(&mut io, &[], &[], &[], &[], &[], &[]);
 		// Try to propagate same transactions for the third time
-		let peer_count3 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+		let peer_count3 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 
 		// 1 message should be send
 		assert_eq!(1, io.packets.len());
@@ -474,7 +493,7 @@ mod tests {
 		let queue = RwLock::new(VecDeque::new());
 		let ss = TestSnapshotService::new();
 		let mut io = TestIo::new(&mut client, &ss, &queue, None);
-		let peer_count = SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+		let peer_count = SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 		io.chain.insert_transaction_to_queue();
 		// New block import should not trigger propagation.
 		// (we only propagate on timeout)
@@ -498,10 +517,10 @@ mod tests {
 		let queue = RwLock::new(VecDeque::new());
 		let ss = TestSnapshotService::new();
 		let mut io = TestIo::new(&mut client, &ss, &queue, None);
-		let peer_count = SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+		let peer_count = SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 		sync.chain_new_blocks(&mut io, &[], &[], &[], &[], &[], &[]);
 		// Try to propagate same transactions for the second time
-		let peer_count2 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+		let peer_count2 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 
 		assert_eq!(0, io.packets.len());
 		assert_eq!(0, peer_count);
@@ -519,7 +538,7 @@ mod tests {
 		// should sent some
 		{
 			let mut io = TestIo::new(&mut client, &ss, &queue, None);
-			let peer_count = SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+			let peer_count = SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 			assert_eq!(1, io.packets.len());
 			assert_eq!(1, peer_count);
 		}
@@ -528,9 +547,9 @@ mod tests {
 		let (peer_count2, peer_count3) = {
 			let mut io = TestIo::new(&mut client, &ss, &queue, None);
 			// Propagate new transactions
-			let peer_count2 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+			let peer_count2 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 			// And now the peer should have all transactions
-			let peer_count3 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+			let peer_count3 = SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 			(peer_count2, peer_count3)
 		};
 
@@ -553,7 +572,7 @@ mod tests {
 		let queue = RwLock::new(VecDeque::new());
 		let ss = TestSnapshotService::new();
 		let mut io = TestIo::new(&mut client, &ss, &queue, None);
-		SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+		SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 
 		let stats = sync.transactions_stats();
 		assert_eq!(stats.len(), 1, "Should maintain stats for single transaction.")
@@ -583,7 +602,7 @@ mod tests {
 		io.peers_info.insert(4, "Parity-Ethereum/v2.7.3-ABCDEFGH".to_owned());
 
 		// and new service transaction is propagated to peers
-		SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+		SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 
 		// peer#2 && peer#4 are receiving service transaction
 		assert!(io.packets.iter().any(|p| p.packet_id == 0x02 && p.recipient == 2)); // TRANSACTIONS_PACKET
@@ -607,7 +626,7 @@ mod tests {
 		io.peers_info.insert(1, "Parity-Ethereum/v2.6".to_owned());
 
 		// and service + non-service transactions are propagated to peers
-		SyncPropagator::propagate_new_transactions(&mut sync, &mut io);
+		SyncPropagator::propagate_new_transactions(&mut sync, &mut io, || true);
 
 		// two separate packets for peer are queued:
 		// 1) with non-service-transaction

ethcore/sync/src/chain/supplier.rs

@@ -27,6 +27,7 @@ use sync_io::SyncIo;
 
 use super::{
 	ChainSync,
+	SyncHandler,
 	RlpResponseResult,
 	PacketDecodeError,
 	BLOCK_BODIES_PACKET,
@@ -47,6 +48,8 @@ use super::{
 	RECEIPTS_PACKET,
 	SNAPSHOT_DATA_PACKET,
 	SNAPSHOT_MANIFEST_PACKET,
+	STATUS_PACKET,
+	TRANSACTIONS_PACKET,
 };
 
 /// The Chain Sync Supplier: answers requests from peers with available data
@@ -56,6 +59,7 @@ impl SyncSupplier {
 	/// Dispatch incoming requests and responses
 	pub fn dispatch_packet(sync: &RwLock<ChainSync>, io: &mut SyncIo, peer: PeerId, packet_id: u8, data: &[u8]) {
 		let rlp = Rlp::new(data);
+
 		let result = match packet_id {
 			GET_BLOCK_BODIES_PACKET => SyncSupplier::return_rlp(io, &rlp, peer,
 				SyncSupplier::return_block_bodies,
@@ -80,9 +84,39 @@ impl SyncSupplier {
 			GET_SNAPSHOT_DATA_PACKET => SyncSupplier::return_rlp(io, &rlp, peer,
 				SyncSupplier::return_snapshot_data,
 				|e| format!("Error sending snapshot data: {:?}", e)),
-			CONSENSUS_DATA_PACKET => ChainSync::on_consensus_packet(io, peer, &rlp),
-			_ => {
+
+			STATUS_PACKET => {
 				sync.write().on_packet(io, peer, packet_id, data);
+				Ok(())
+			},
+			// Packets that require the peer to be confirmed
+			_ => {
+				if !sync.read().peers.contains_key(&peer) {
+					debug!(target:"sync", "Unexpected packet {} from unregistered peer: {}:{}", packet_id, peer, io.peer_info(peer));
+					return;
+				}
+				debug!(target: "sync", "{} -> Dispatching packet: {}", peer, packet_id);
+
+				match packet_id {
+					CONSENSUS_DATA_PACKET => {
+						SyncHandler::on_consensus_packet(io, peer, &rlp)
+					},
+					TRANSACTIONS_PACKET => {
+						let res = {
+							let sync_ro = sync.read();
+							SyncHandler::on_peer_transactions(&*sync_ro, io, peer, &rlp)
+						};
+						if res.is_err() {
+							// peer sent invalid data, disconnect.
+							io.disable_peer(peer);
+							sync.write().deactivate_peer(io, peer);
+						}
+					},
+					_ => {
+						sync.write().on_packet(io, peer, packet_id, data);
+					}
+				}
+
 				Ok(())
 			}
 		};
@@ -226,7 +260,8 @@ impl SyncSupplier {
 		let mut added_receipts = 0usize;
 		let mut data = Bytes::new();
 		for i in 0..count {
-			if let Some(mut receipts_bytes) = io.chain().encoded_block_receipts(&rlp.val_at::<H256>(i)?) {
+			if let Some(receipts) = io.chain().block_receipts(&rlp.val_at::<H256>(i)?) {
+				let mut receipts_bytes = ::rlp::encode(&receipts);
 				data.append(&mut receipts_bytes);
 				added_receipts += receipts_bytes.len();
 				added_headers += 1;
@@ -403,7 +438,7 @@ mod test {
 
 		io.sender = Some(2usize);
 
-		ChainSync::dispatch_packet(&RwLock::new(sync), &mut io, 0usize, GET_NODE_DATA_PACKET, &node_request);
+		SyncSupplier::dispatch_packet(&RwLock::new(sync), &mut io, 0usize, GET_NODE_DATA_PACKET, &node_request);
 		assert_eq!(1, io.packets.len());
 	}
 
@@ -445,7 +480,7 @@ mod test {
 		assert_eq!(603, rlp_result.unwrap().1.out().len());
 
 		io.sender = Some(2usize);
-		ChainSync::dispatch_packet(&RwLock::new(sync), &mut io, 0usize, GET_RECEIPTS_PACKET, &receipts_request);
+		SyncSupplier::dispatch_packet(&RwLock::new(sync), &mut io, 0usize, GET_RECEIPTS_PACKET, &receipts_request);
 		assert_eq!(1, io.packets.len());
 	}
 }

ethcore/sync/src/light_sync/mod.rs

@@ -34,6 +34,7 @@
 
 use std::collections::{HashMap, HashSet};
 use std::mem;
+use std::ops::Deref;
 use std::sync::Arc;
 use std::time::{Instant, Duration};
 
@@ -213,6 +214,44 @@ enum SyncState {
 	Rounds(SyncRound),
 }
 
+/// A wrapper around the SyncState that makes sure to
+/// update the giving reference to `is_idle`
+#[derive(Debug)]
+struct SyncStateWrapper {
+	state: SyncState,
+}
+
+impl SyncStateWrapper {
+	/// Create a new wrapper for SyncState::Idle
+	pub fn idle() -> Self {
+		SyncStateWrapper {
+			state: SyncState::Idle,
+		}
+	}
+
+	/// Set the new state's value, making sure `is_idle` gets updated
+	pub fn set(&mut self, state: SyncState, is_idle_handle: &mut bool) {
+		*is_idle_handle = match state {
+			SyncState::Idle => true,
+			_ => false,
+		};
+		self.state = state;
+	}
+
+	/// Returns the internal state's value
+	pub fn into_inner(self) -> SyncState {
+		self.state
+	}
+}
+
+impl Deref for SyncStateWrapper {
+	type Target = SyncState;
+
+	fn deref(&self) -> &SyncState {
+		&self.state
+	}
+}
+
 struct ResponseCtx<'a> {
 	peer: PeerId,
 	req_id: ReqId,
@@ -235,7 +274,9 @@ pub struct LightSync<L: AsLightClient> {
 	pending_reqs: Mutex<HashMap<ReqId, PendingReq>>, // requests from this handler
 	client: Arc<L>,
 	rng: Mutex<OsRng>,
-	state: Mutex<SyncState>,
+	state: Mutex<SyncStateWrapper>,
+	// We duplicate this state tracking to avoid deadlocks in `is_major_importing`.
+	is_idle: Mutex<bool>,
 }
 
 #[derive(Debug, Clone)]
@@ -309,16 +350,17 @@ impl<L: AsLightClient + Send + Sync> Handler for LightSync<L> {
 
 		if new_best.is_none() {
 			debug!(target: "sync", "No peers remain. Reverting to idle");
-			*self.state.lock() = SyncState::Idle;
+			self.set_state(&mut self.state.lock(), SyncState::Idle);
 		} else {
 			let mut state = self.state.lock();
 
-			*state = match mem::replace(&mut *state, SyncState::Idle) {
+			let next_state = match mem::replace(&mut *state, SyncStateWrapper::idle()).into_inner() {
 				SyncState::Idle => SyncState::Idle,
 				SyncState::AncestorSearch(search) =>
 					SyncState::AncestorSearch(search.requests_abandoned(unfulfilled)),
 				SyncState::Rounds(round) => SyncState::Rounds(round.requests_abandoned(unfulfilled)),
 			};
+			self.set_state(&mut state, next_state);
 		}
 
 		self.maintain_sync(ctx.as_basic());
@@ -390,12 +432,13 @@ impl<L: AsLightClient + Send + Sync> Handler for LightSync<L> {
 				data: headers,
 			};
 
-			*state = match mem::replace(&mut *state, SyncState::Idle) {
+			let next_state = match mem::replace(&mut *state, SyncStateWrapper::idle()).into_inner() {
 				SyncState::Idle => SyncState::Idle,
 				SyncState::AncestorSearch(search) =>
 					SyncState::AncestorSearch(search.process_response(&ctx, &*self.client)),
 				SyncState::Rounds(round) => SyncState::Rounds(round.process_response(&ctx)),
 			};
+			self.set_state(&mut state, next_state);
 		}
 
 		self.maintain_sync(ctx.as_basic());
@@ -408,12 +451,18 @@ impl<L: AsLightClient + Send + Sync> Handler for LightSync<L> {
 
 // private helpers
 impl<L: AsLightClient> LightSync<L> {
+	/// Sets the LightSync's state, and update
+	/// `is_idle`
+	fn set_state(&self, state: &mut SyncStateWrapper, next_state: SyncState) {
+		state.set(next_state, &mut self.is_idle.lock());
+	}
+
 	// Begins a search for the common ancestor and our best block.
 	// does not lock state, instead has a mutable reference to it passed.
-	fn begin_search(&self, state: &mut SyncState) {
+	fn begin_search(&self, state: &mut SyncStateWrapper) {
 		if let None =  *self.best_seen.lock() {
 			// no peers.
-			*state = SyncState::Idle;
+			self.set_state(state, SyncState::Idle);
 			return;
 		}
 
@@ -422,7 +471,8 @@ impl<L: AsLightClient> LightSync<L> {
 
 		trace!(target: "sync", "Beginning search for common ancestor from {:?}",
 			(chain_info.best_block_number, chain_info.best_block_hash));
-		*state = SyncState::AncestorSearch(AncestorSearch::begin(chain_info.best_block_number));
+		let next_state = SyncState::AncestorSearch(AncestorSearch::begin(chain_info.best_block_number));
+		self.set_state(state, next_state);
 	}
 
 	// handles request dispatch, block import, state machine transitions, and timeouts.
@@ -435,7 +485,7 @@ impl<L: AsLightClient> LightSync<L> {
 		let chain_info = client.chain_info();
 
 		let mut state = self.state.lock();
-		debug!(target: "sync", "Maintaining sync ({:?})", &*state);
+		debug!(target: "sync", "Maintaining sync ({:?})", **state);
 
 		// drain any pending blocks into the queue.
 		{
@@ -445,11 +495,12 @@ impl<L: AsLightClient> LightSync<L> {
 			loop {
 				if client.queue_info().is_full() { break }
 
-				*state = match mem::replace(&mut *state, SyncState::Idle) {
+				let next_state = match mem::replace(&mut *state, SyncStateWrapper::idle()).into_inner() {
 					SyncState::Rounds(round)
 						=> SyncState::Rounds(round.drain(&mut sink, Some(DRAIN_AMOUNT))),
 					other => other,
 				};
+				self.set_state(&mut state, next_state);
 
 				if sink.is_empty() { break }
 				trace!(target: "sync", "Drained {} headers to import", sink.len());
@@ -483,15 +534,15 @@ impl<L: AsLightClient> LightSync<L> {
 					let network_score = other.as_ref().map(|target| target.head_td);
 					trace!(target: "sync", "No target to sync to. Network score: {:?}, Local score: {:?}",
 						network_score, best_td);
-					*state = SyncState::Idle;
+					self.set_state(&mut state, SyncState::Idle);
 					return;
 				}
 			};
 
-			match mem::replace(&mut *state, SyncState::Idle) {
+			match mem::replace(&mut *state, SyncStateWrapper::idle()).into_inner() {
 				SyncState::Rounds(SyncRound::Abort(reason, remaining)) => {
 					if remaining.len() > 0 {
-						*state = SyncState::Rounds(SyncRound::Abort(reason, remaining));
+						self.set_state(&mut state, SyncState::Rounds(SyncRound::Abort(reason, remaining)));
 						return;
 					}
 
@@ -505,7 +556,7 @@ impl<L: AsLightClient> LightSync<L> {
 						AbortReason::NoResponses => {}
 						AbortReason::TargetReached => {
 							debug!(target: "sync", "Sync target reached. Going idle");
-							*state = SyncState::Idle;
+							self.set_state(&mut state, SyncState::Idle);
 							return;
 						}
 					}
@@ -514,15 +565,15 @@ impl<L: AsLightClient> LightSync<L> {
 					self.begin_search(&mut state);
 				}
 				SyncState::AncestorSearch(AncestorSearch::FoundCommon(num, hash)) => {
-					*state = SyncState::Rounds(SyncRound::begin((num, hash), sync_target));
+					self.set_state(&mut state, SyncState::Rounds(SyncRound::begin((num, hash), sync_target)));
 				}
 				SyncState::AncestorSearch(AncestorSearch::Genesis) => {
 					// Same here.
 					let g_hash = chain_info.genesis_hash;
-					*state = SyncState::Rounds(SyncRound::begin((0, g_hash), sync_target));
+					self.set_state(&mut state, SyncState::Rounds(SyncRound::begin((0, g_hash), sync_target)));
 				}
 				SyncState::Idle => self.begin_search(&mut state),
-				other => *state = other, // restore displaced state.
+				other => self.set_state(&mut state, other), // restore displaced state.
 			}
 		}
 
@@ -543,12 +594,13 @@ impl<L: AsLightClient> LightSync<L> {
 				}
 				drop(pending_reqs);
 
-				*state = match mem::replace(&mut *state, SyncState::Idle) {
+				let next_state = match mem::replace(&mut *state, SyncStateWrapper::idle()).into_inner() {
 					SyncState::Idle => SyncState::Idle,
 					SyncState::AncestorSearch(search) =>
 						SyncState::AncestorSearch(search.requests_abandoned(&unfulfilled)),
 					SyncState::Rounds(round) => SyncState::Rounds(round.requests_abandoned(&unfulfilled)),
 				};
+				self.set_state(&mut state, next_state);
 			}
 		}
 
@@ -605,34 +657,14 @@ impl<L: AsLightClient> LightSync<L> {
 				None
 			};
 
-			*state = match mem::replace(&mut *state, SyncState::Idle) {
+			let next_state = match mem::replace(&mut *state, SyncStateWrapper::idle()).into_inner() {
 				SyncState::Rounds(round) =>
 					SyncState::Rounds(round.dispatch_requests(dispatcher)),
 				SyncState::AncestorSearch(search) =>
 					SyncState::AncestorSearch(search.dispatch_request(dispatcher)),
 				other => other,
 			};
-		}
-	}
-
-	fn is_major_importing_do_wait(&self, wait: bool) -> bool {
-		const EMPTY_QUEUE: usize = 3;
-
-		if self.client.as_light_client().queue_info().unverified_queue_size > EMPTY_QUEUE {
-			return true;
-		}
-		let mg_state = if wait {
-			self.state.lock()
-		} else {
-			if let Some(mg_state) = self.state.try_lock() {
-				mg_state
-			} else {
-				return false;
-			}
-		};
-		match *mg_state {
-			SyncState::Idle => false,
-			_ => true,
+			self.set_state(&mut state, next_state);
 		}
 	}
 }
@@ -651,7 +683,8 @@ impl<L: AsLightClient> LightSync<L> {
 			pending_reqs: Mutex::new(HashMap::new()),
 			client: client,
 			rng: Mutex::new(OsRng::new()?),
-			state: Mutex::new(SyncState::Idle),
+			state: Mutex::new(SyncStateWrapper::idle()),
+			is_idle: Mutex::new(true),
 		})
 	}
 }
@@ -666,9 +699,6 @@ pub trait SyncInfo {
 
 	/// Whether major sync is underway.
 	fn is_major_importing(&self) -> bool;
-
-	/// Whether major sync is underway, skipping some synchronization.
-	fn is_major_importing_no_sync(&self) -> bool;
 }
 
 impl<L: AsLightClient> SyncInfo for LightSync<L> {
@@ -681,11 +711,13 @@ impl<L: AsLightClient> SyncInfo for LightSync<L> {
 	}
 
 	fn is_major_importing(&self) -> bool {
-		self.is_major_importing_do_wait(true)
-	}
+		const EMPTY_QUEUE: usize = 3;
 
-	fn is_major_importing_no_sync(&self) -> bool {
-		self.is_major_importing_do_wait(false)
+		let queue_info = self.client.as_light_client().queue_info();
+		let is_verifying = queue_info.unverified_queue_size + queue_info.verified_queue_size > EMPTY_QUEUE;
+		let is_syncing = !*self.is_idle.lock();
+
+		is_verifying || is_syncing
 	}
 
 }

ethcore/sync/src/sync_io.rs

@@ -52,7 +52,7 @@ pub trait SyncIo {
 	fn protocol_version(&self, protocol: &ProtocolId, peer_id: PeerId) -> u8;
 	/// Returns if the chain block queue empty
 	fn is_chain_queue_empty(&self) -> bool {
-		self.chain().queue_info().is_empty()
+		self.chain().is_queue_empty()
 	}
 	/// Check if the session is expired
 	fn is_expired(&self) -> bool;

ethcore/sync/src/tests/helpers.rs

@@ -33,7 +33,7 @@ use ethcore::test_helpers;
 use sync_io::SyncIo;
 use io::{IoChannel, IoContext, IoHandler};
 use api::WARP_SYNC_PROTOCOL_ID;
-use chain::{ChainSync, ETH_PROTOCOL_VERSION_63, PAR_PROTOCOL_VERSION_3, PRIVATE_TRANSACTION_PACKET, SIGNED_PRIVATE_TRANSACTION_PACKET};
+use chain::{ChainSync, ETH_PROTOCOL_VERSION_63, PAR_PROTOCOL_VERSION_3, PRIVATE_TRANSACTION_PACKET, SIGNED_PRIVATE_TRANSACTION_PACKET, SyncSupplier};
 use SyncConfig;
 use private_tx::SimplePrivateTxHandler;
 
@@ -271,7 +271,7 @@ impl<C: FlushingBlockChainClient> Peer for EthPeer<C> {
 
 	fn receive_message(&self, from: PeerId, msg: TestPacket) -> HashSet<PeerId> {
 		let mut io = TestIo::new(&*self.chain, &self.snapshot_service, &self.queue, Some(from));
-		ChainSync::dispatch_packet(&self.sync, &mut io, from, msg.packet_id, &msg.data);
+		SyncSupplier::dispatch_packet(&self.sync, &mut io, from, msg.packet_id, &msg.data);
 		self.chain.flush();
 		io.to_disconnect.clone()
 	}
@@ -286,10 +286,12 @@ impl<C: FlushingBlockChainClient> Peer for EthPeer<C> {
 	}
 
 	fn sync_step(&self) {
+		let mut io = TestIo::new(&*self.chain, &self.snapshot_service, &self.queue, None);
 		self.chain.flush();
-		self.sync.write().maintain_peers(&mut TestIo::new(&*self.chain, &self.snapshot_service, &self.queue, None));
-		self.sync.write().maintain_sync(&mut TestIo::new(&*self.chain, &self.snapshot_service, &self.queue, None));
-		self.sync.write().propagate_new_transactions(&mut TestIo::new(&*self.chain, &self.snapshot_service, &self.queue, None));
+		self.sync.write().maintain_peers(&mut io);
+		self.sync.write().maintain_sync(&mut io);
+		self.sync.write().continue_sync(&mut io);
+		self.sync.write().propagate_new_transactions(&mut io);
 	}
 
 	fn restart_sync(&self) {

ethcore/sync/src/transactions_stats.rs

@@ -15,6 +15,7 @@
 // along with Parity.  If not, see <http://www.gnu.org/licenses/>.
 
 use api::TransactionStats;
+use std::hash::BuildHasher;
 use std::collections::{HashSet, HashMap};
 use ethereum_types::{H256, H512};
 use fastmap::H256FastMap;
@@ -74,7 +75,7 @@ impl TransactionsStats {
 	}
 
 	/// Retains only transactions present in given `HashSet`.
-	pub fn retain(&mut self, hashes: &HashSet<H256>) {
+	pub fn retain<S: BuildHasher>(&mut self, hashes: &HashSet<H256, S>) {
 		let to_remove = self.pending_transactions.keys()
 			.filter(|hash| !hashes.contains(hash))
 			.cloned()

evmbin/src/display/json.rs

@@ -55,6 +55,22 @@ impl Informant {
 			Self::with_informant_in_depth(informant.subinfos.last_mut().expect("prepare/done_trace are not balanced"), depth - 1, f);
 		}
 	}
+
+	fn informant_trace(informant: &Informant, gas_used: U256) -> String {
+		let info = ::evm::Instruction::from_u8(informant.instruction).map(|i| i.info());
+
+		json!({
+			"pc": informant.pc,
+			"op": informant.instruction,
+			"opName": info.map(|i| i.name).unwrap_or(""),
+			"gas": format!("{:#x}", gas_used.saturating_add(informant.gas_cost)),
+			"gasCost": format!("{:#x}", informant.gas_cost),
+			"memory": format!("0x{}", informant.memory.to_hex()),
+			"stack": informant.stack,
+			"storage": informant.storage,
+			"depth": informant.depth,
+		}).to_string()
+	}
 }
 
 impl vm::Informant for Informant {
@@ -125,22 +141,11 @@ impl trace::VMTracer for Informant {
 	fn trace_executed(&mut self, gas_used: U256, stack_push: &[U256], mem: &[u8]) {
 		let subdepth = self.subdepth;
 		Self::with_informant_in_depth(self, subdepth, |informant: &mut Informant| {
-			let mem_diff = informant.mem_written.clone().map(|(o, s)| (o, &(mem[o..o+s])));
 			let store_diff = informant.store_written.clone();
 			let info = ::evm::Instruction::from_u8(informant.instruction).map(|i| i.info());
 
-			let trace = json!({
-				"pc": informant.pc,
-				"op": informant.instruction,
-				"opName": info.map(|i| i.name).unwrap_or(""),
-				"gas": format!("{:#x}", gas_used.saturating_add(informant.gas_cost)),
-				"gasCost": format!("{:#x}", informant.gas_cost),
-				"memory": format!("0x{}", informant.memory.to_hex()),
-				"stack": informant.stack,
-				"storage": informant.storage,
-				"depth": informant.depth,
-			});
-			informant.traces.push(trace.to_string());
+			let trace = Self::informant_trace(informant, gas_used);
+			informant.traces.push(trace);
 
 			informant.unmatched = false;
 			informant.gas_used = gas_used;
@@ -151,11 +156,11 @@ impl trace::VMTracer for Informant {
 			informant.stack.extend_from_slice(stack_push);
 
 			// TODO [ToDr] Align memory?
-			if let Some((pos, data)) = mem_diff {
-				if informant.memory.len() < (pos + data.len()) {
-					informant.memory.resize(pos + data.len(), 0);
+			if let Some((pos, size)) = informant.mem_written.clone() {
+				if informant.memory.len() < (pos + size) {
+					informant.memory.resize(pos + size, 0);
 				}
-				informant.memory[pos..pos + data.len()].copy_from_slice(data);
+				informant.memory[pos..(pos + size)].copy_from_slice(&mem[pos..(pos + size)]);
 			}
 
 			if let Some((pos, val)) = store_diff {
@@ -195,7 +200,12 @@ impl trace::VMTracer for Informant {
 			// print last line with final state:
 			self.gas_cost = 0.into();
 			let gas_used = self.gas_used;
-			self.trace_executed(gas_used, &[], &[]);
+			let subdepth = self.subdepth;
+
+			Self::with_informant_in_depth(&mut self, subdepth, |informant: &mut Informant| {
+				let trace = Self::informant_trace(informant, gas_used);
+				informant.traces.push(trace);
+			});
 		} else if !self.subtraces.is_empty() {
 			self.traces.extend(mem::replace(&mut self.subtraces, vec![]));
 		}
@@ -280,6 +290,17 @@ mod tests {
 {"pc":0,"op":248,"opName":"","gas":"0xffff","gasCost":"0x0","memory":"0x","stack":[],"storage":{},"depth":1}
 			"#,
 		);
+
+		run_test(
+			Informant::default(),
+			&compare_json,
+			"5A51",
+			0xfffff,
+			r#"
+{"depth":1,"gas":"0xfffff","gasCost":"0x2","memory":"0x","op":90,"opName":"GAS","pc":0,"stack":[],"storage":{}}
+{"depth":1,"gas":"0xffffd","gasCost":"0x0","memory":"0x","op":81,"opName":"MLOAD","pc":1,"stack":["0xffffd"],"storage":{}}
+			"#,
+		);
 	}
 
 	#[test]

json/src/blockchain/blockchain.rs

@@ -23,6 +23,21 @@ use blockchain::header::Header;
 use blockchain::block::Block;
 use spec::{ForkSpec, Genesis, Seal, Ethereum};
 
+/// Json Block test possible engine kind.
+#[derive(Debug, PartialEq, Deserialize)]
+pub enum Engine {
+	/// Default (old) behaviour.
+	Ethash,
+	/// No check of block's difficulty and nonce for tests.
+	NoProof,
+}
+
+impl Default for Engine {
+	fn default() -> Self {
+		Engine::Ethash
+	}
+}
+
 /// Blockchain deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
 pub struct BlockChain {
@@ -45,6 +60,10 @@ pub struct BlockChain {
 	pub best_block: H256,
 	/// Network.
 	pub network: ForkSpec,
+	#[serde(default)]
+	#[serde(rename="sealEngine")]
+	/// Engine
+	pub engine: Engine,
 }
 
 impl BlockChain {

json/src/blockchain/mod.rs

@@ -27,6 +27,7 @@ pub mod test;
 pub use self::account::Account;
 pub use self::block::Block;
 pub use self::blockchain::BlockChain;
+pub use self::blockchain::Engine;
 pub use self::header::Header;
 pub use self::state::State;
 pub use self::test::Test;

json/src/spec/account.rs

@@ -23,6 +23,7 @@ use spec::builtin::Builtin;
 
 /// Spec account.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct Account {
 	/// Builtin contract.
 	pub builtin: Option<Builtin>,

json/src/spec/authority_round.rs

@@ -23,6 +23,7 @@ use super::ValidatorSet;
 
 /// Authority params deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct AuthorityRoundParams {
 	/// Block duration, in seconds.
 	#[serde(rename="stepDuration")]
@@ -71,6 +72,7 @@ pub struct AuthorityRoundParams {
 
 /// Authority engine deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct AuthorityRound {
 	/// Ethash params.
 	pub params: AuthorityRoundParams,

json/src/spec/basic_authority.rs

@@ -21,6 +21,7 @@ use super::ValidatorSet;
 
 /// Authority params deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct BasicAuthorityParams {
 	/// Block duration.
 	#[serde(rename="durationLimit")]
@@ -31,6 +32,7 @@ pub struct BasicAuthorityParams {
 
 /// Authority engine deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct BasicAuthority {
 	/// Ethash params.
 	pub params: BasicAuthorityParams,

json/src/spec/builtin.rs

@@ -20,6 +20,7 @@ use uint::Uint;
 
 /// Linear pricing.
 #[derive(Debug, PartialEq, Deserialize, Clone)]
+#[serde(deny_unknown_fields)]
 pub struct Linear {
 	/// Base price.
 	pub base: usize,
@@ -29,6 +30,7 @@ pub struct Linear {
 
 /// Pricing for modular exponentiation.
 #[derive(Debug, PartialEq, Deserialize, Clone)]
+#[serde(deny_unknown_fields)]
 pub struct Modexp {
 	/// Price divisor.
 	pub divisor: usize,
@@ -36,6 +38,7 @@ pub struct Modexp {
 
 /// Pricing for alt_bn128_pairing.
 #[derive(Debug, PartialEq, Deserialize, Clone)]
+#[serde(deny_unknown_fields)]
 pub struct AltBn128Pairing {
 	/// Base price.
 	pub base: usize,
@@ -45,6 +48,7 @@ pub struct AltBn128Pairing {
 
 /// Pricing variants.
 #[derive(Debug, PartialEq, Deserialize, Clone)]
+#[serde(deny_unknown_fields)]
 pub enum Pricing {
 	/// Linear pricing.
 	#[serde(rename="linear")]
@@ -59,6 +63,7 @@ pub enum Pricing {
 
 /// Spec builtin.
 #[derive(Debug, PartialEq, Deserialize, Clone)]
+#[serde(deny_unknown_fields)]
 pub struct Builtin {
 	/// Builtin name.
 	pub name: String,

json/src/spec/engine.rs

@@ -20,6 +20,7 @@ use super::{Ethash, BasicAuthority, AuthorityRound, Tendermint, NullEngine, Inst
 
 /// Engine deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub enum Engine {
 	/// Null engine.
 	#[serde(rename="null")]
@@ -28,6 +29,7 @@ pub enum Engine {
 	#[serde(rename="instantSeal")]
 	InstantSeal(Option<InstantSeal>),
 	/// Ethash engine.
+	#[serde(rename = "Ethash")]
 	Ethash(Ethash),
 	/// BasicAuthority engine.
 	#[serde(rename="basicAuthority")]
@@ -88,7 +90,6 @@ mod tests {
 					"minimumDifficulty": "0x020000",
 					"difficultyBoundDivisor": "0x0800",
 					"durationLimit": "0x0d",
-					"registrar" : "0xc6d9d2cd449a754c494264e1809c50e34d64562b",
 					"homesteadTransition" : "0x",
 					"daoHardforkTransition": "0xffffffffffffffff",
 					"daoHardforkBeneficiary": "0x0000000000000000000000000000000000000000",

json/src/spec/ethash.rs

@@ -23,6 +23,7 @@ use hash::Address;
 
 /// Deserializable doppelganger of block rewards for EthashParams
 #[derive(Clone, Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 #[serde(untagged)]
 pub enum BlockReward {
 	Single(Uint),
@@ -31,6 +32,7 @@ pub enum BlockReward {
 
 /// Deserializable doppelganger of EthashParams.
 #[derive(Clone, Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct EthashParams {
 	/// See main EthashParams docs.
 	#[serde(rename="minimumDifficulty")]
@@ -119,6 +121,7 @@ pub struct EthashParams {
 
 /// Ethash engine deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct Ethash {
 	/// Ethash params.
 	pub params: EthashParams,

json/src/spec/genesis.rs

@@ -23,6 +23,7 @@ use spec::Seal;
 
 /// Spec genesis.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct Genesis {
 	/// Seal.
 	pub seal: Seal,
@@ -70,7 +71,6 @@ mod tests {
 	#[test]
 	fn genesis_deserialization() {
 		let s = r#"{
-			"nonce": "0x0000000000000042",
 			"difficulty": "0x400000000",
 			"seal": {
 				"ethereum": {

json/src/spec/hardcoded_sync.rs

@@ -21,6 +21,7 @@ use uint::Uint;
 
 /// Spec hardcoded sync.
 #[derive(Debug, PartialEq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct HardcodedSync {
 	/// Hexadecimal of the RLP encoding of the header of the block to start synchronization from.
 	pub header: String,
@@ -28,7 +29,7 @@ pub struct HardcodedSync {
 	#[serde(rename="totalDifficulty")]
 	pub total_difficulty: Uint,
 	/// Ordered trie roots of blocks before and including `header`.
-	#[serde(rename="CHTs")]
+	#[serde(rename = "CHTs")]
 	pub chts: Vec<H256>,
 }
 

json/src/spec/instant_seal.rs

@@ -18,6 +18,7 @@
 
 /// Instant seal engine params deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct InstantSealParams {
 	/// Whether to enable millisecond timestamp.
 	#[serde(rename="millisecondTimestamp")]
@@ -27,6 +28,7 @@ pub struct InstantSealParams {
 
 /// Instant seal engine descriptor.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct InstantSeal {
 	/// Instant seal parameters.
 	pub params: InstantSealParams,

json/src/spec/null_engine.rs

@@ -20,6 +20,7 @@ use uint::Uint;
 
 /// Authority params deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct NullEngineParams {
 	/// Block reward.
 	#[serde(rename="blockReward")]
@@ -28,6 +29,7 @@ pub struct NullEngineParams {
 
 /// Null engine descriptor
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct NullEngine {
 	/// Ethash params.
 	pub params: NullEngineParams,

json/src/spec/params.rs

@@ -22,6 +22,7 @@ use bytes::Bytes;
 
 /// Spec params.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct Params {
 	/// Account start nonce, defaults to 0.
 	#[serde(rename="accountStartNonce")]

json/src/spec/seal.rs

@@ -22,6 +22,7 @@ use bytes::Bytes;
 
 /// Ethereum seal.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct Ethereum {
 	/// Seal nonce.
 	pub nonce: H64,
@@ -32,6 +33,7 @@ pub struct Ethereum {
 
 /// AuthorityRound seal.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct AuthorityRoundSeal {
 	/// Seal step.
 	pub step: Uint,
@@ -41,6 +43,7 @@ pub struct AuthorityRoundSeal {
 
 /// Tendermint seal.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct TendermintSeal {
 	/// Seal round.
 	pub round: Uint,
@@ -52,6 +55,7 @@ pub struct TendermintSeal {
 
 /// Seal variants.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub enum Seal {
 	/// Ethereum seal.
 	#[serde(rename="ethereum")]

json/src/spec/spec.rs

@@ -38,6 +38,7 @@ pub enum ForkSpec {
 
 /// Spec deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct Spec {
 	/// Spec name.
 	pub name: String,
@@ -71,6 +72,71 @@ mod tests {
 	use serde_json;
 	use spec::spec::Spec;
 
+	#[test]
+	fn should_error_on_unknown_fields() {
+		let s = r#"{
+	"name": "Morden",
+	"dataDir": "morden",
+	"engine": {
+		"Ethash": {
+			"params": {
+				"minimumDifficulty": "0x020000",
+				"difficultyBoundDivisor": "0x0800",
+				"durationLimit": "0x0d",
+				"homesteadTransition" : "0x",
+				"daoHardforkTransition": "0xffffffffffffffff",
+				"daoHardforkBeneficiary": "0x0000000000000000000000000000000000000000",
+				"daoHardforkAccounts": []
+			}
+		}
+	},
+	"params": {
+		"accountStartNonce": "0x0100000",
+		"maximumExtraDataSize": "0x20",
+		"minGasLimit": "0x1388",
+		"networkID" : "0x2",
+		"forkBlock": "0xffffffffffffffff",
+		"forkCanonHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
+		"gasLimitBoundDivisor": "0x20",
+		"unknownField": "0x0"
+	},
+	"genesis": {
+		"seal": {
+			"ethereum": {
+				"mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
+				"nonce": "0x00006d6f7264656e"
+			}
+		},
+		"difficulty": "0x20000",
+		"author": "0x0000000000000000000000000000000000000000",
+		"timestamp": "0x00",
+		"parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
+		"extraData": "0x",
+		"gasLimit": "0x2fefd8"
+	},
+	"nodes": [
+		"enode://b1217cbaa440e35ed471157123fe468e19e8b5ad5bedb4b1fdbcbdab6fb2f5ed3e95dd9c24a22a79fdb2352204cea207df27d92bfd21bfd41545e8b16f637499@104.44.138.37:30303"
+	],
+	"accounts": {
+		"0000000000000000000000000000000000000001": { "balance": "1", "nonce": "1048576", "builtin": { "name": "ecrecover", "pricing": { "linear": { "base": 3000, "word": 0 } } } },
+		"0000000000000000000000000000000000000002": { "balance": "1", "nonce": "1048576", "builtin": { "name": "sha256", "pricing": { "linear": { "base": 60, "word": 12 } } } },
+		"0000000000000000000000000000000000000003": { "balance": "1", "nonce": "1048576", "builtin": { "name": "ripemd160", "pricing": { "linear": { "base": 600, "word": 120 } } } },
+		"0000000000000000000000000000000000000004": { "balance": "1", "nonce": "1048576", "builtin": { "name": "identity", "pricing": { "linear": { "base": 15, "word": 3 } } } },
+		"102e61f5d8f9bc71d0ad4a084df4e65e05ce0e1c": { "balance": "1606938044258990275541962092341162602522202993782792835301376", "nonce": "1048576" }
+	},
+	"hardcodedSync": {
+		"header": "f901f9a0d405da4e66f1445d455195229624e133f5baafe72b5cf7b3c36c12c8146e98b7a01dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347948888f1f195afa192cfee860698584c030f4c9db1a05fb2b4bfdef7b314451cb138a534d225c922fc0e5fbe25e451142732c3e25c25a088d2ec6b9860aae1a2c3b299f72b6a5d70d7f7ba4722c78f2c49ba96273c2158a007c6fdfa8eea7e86b81f5b0fc0f78f90cc19f4aa60d323151e0cac660199e9a1b90100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008302008003832fefba82524d84568e932a80a0a0349d8c3df71f1a48a9df7d03fd5f14aeee7d91332c009ecaff0a71ead405bd88ab4e252a7e8c2a23",
+		"totalDifficulty": "0x400000000",
+		"CHTs": [
+			"0x11bbe8db4e347b4e8c937c1c8370e4b5ed33adb3db69cbdb7a38e1e50b1b82fa",
+			"0xd7f8974fb5ac78d9ac099b9ad5018bedc2ce0a72dad1827a1709da30580f0544"
+		]
+	}
+		}"#;
+		let result: Result<Spec, _> = serde_json::from_str(s);
+		assert!(result.is_err());
+	}
+
 	#[test]
 	fn spec_deserialization() {
 		let s = r#"{
@@ -91,7 +157,6 @@ mod tests {
 	},
 	"params": {
 		"accountStartNonce": "0x0100000",
-		"homesteadTransition": "0x789b0",
 		"maximumExtraDataSize": "0x20",
 		"minGasLimit": "0x1388",
 		"networkID" : "0x2",

json/src/spec/state.rs

@@ -23,6 +23,7 @@ use spec::{Account, Builtin};
 
 /// Blockchain test state deserializer.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct State(BTreeMap<Address, Account>);
 
 impl State {

json/src/spec/tendermint.rs

@@ -21,6 +21,7 @@ use super::ValidatorSet;
 
 /// Tendermint params deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct TendermintParams {
 	/// Valid validators.
 	pub validators: ValidatorSet,
@@ -43,6 +44,7 @@ pub struct TendermintParams {
 
 /// Tendermint engine deserialization.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub struct Tendermint {
 	/// Ethash params.
 	pub params: TendermintParams,

json/src/spec/validator_set.rs

@@ -22,6 +22,7 @@ use hash::Address;
 
 /// Different ways of specifying validators.
 #[derive(Debug, PartialEq, Deserialize)]
+#[serde(deny_unknown_fields)]
 pub enum ValidatorSet {
 	/// A simple list of authorities.
 	#[serde(rename="list")]

miner/src/pool/listener.rs

@@ -50,6 +50,10 @@ impl Notifier {
 
 	/// Notify listeners about all currently pending transactions.
 	pub fn notify(&mut self) {
+		if self.pending.is_empty() {
+			return;
+		}
+
 		for l in &self.listeners {
 			(l)(&self.pending);
 		}

parity/cli/mod.rs

@@ -467,6 +467,10 @@ usage! {
 			"--no-jsonrpc",
 			"Disable the HTTP JSON-RPC API server.",
 
+			FLAG flag_jsonrpc_experimental: (bool) = false, or |c: &Config| c.rpc.as_ref()?.experimental_rpcs.clone(),
+			"--jsonrpc-experimental",
+			"Enable experimental RPCs. Enable to have access to methods from unfinalised EIPs in all namespaces",
+
 			ARG arg_jsonrpc_port: (u16) = 8545u16, or |c: &Config| c.rpc.as_ref()?.port.clone(),
 			"--jsonrpc-port=[PORT]",
 			"Specify the port portion of the HTTP JSON-RPC API server.",
@@ -1141,7 +1145,7 @@ struct Operating {
 	no_persistent_txqueue: Option<bool>,
 	no_hardcoded_sync: Option<bool>,
 
-	#[serde(rename="public_node")]
+	#[serde(rename = "public_node")]
 	_legacy_public_node: Option<bool>,
 }
 
@@ -1173,15 +1177,15 @@ struct PrivateTransactions {
 struct Ui {
 	path: Option<String>,
 
-	#[serde(rename="force")]
+	#[serde(rename = "force")]
 	_legacy_force: Option<bool>,
-	#[serde(rename="disable")]
+	#[serde(rename = "disable")]
 	_legacy_disable: Option<bool>,
-	#[serde(rename="port")]
+	#[serde(rename = "port")]
 	_legacy_port: Option<u16>,
-	#[serde(rename="interface")]
+	#[serde(rename = "interface")]
 	_legacy_interface: Option<String>,
-	#[serde(rename="hosts")]
+	#[serde(rename = "hosts")]
 	_legacy_hosts: Option<Vec<String>>,
 }
 
@@ -1219,6 +1223,7 @@ struct Rpc {
 	server_threads: Option<usize>,
 	processing_threads: Option<usize>,
 	max_payload: Option<usize>,
+	experimental_rpcs: Option<bool>,
 }
 
 #[derive(Default, Debug, PartialEq, Deserialize)]
@@ -1244,21 +1249,21 @@ struct Ipc {
 #[derive(Default, Debug, PartialEq, Deserialize)]
 #[serde(deny_unknown_fields)]
 struct Dapps {
-	#[serde(rename="disable")]
+	#[serde(rename = "disable")]
 	_legacy_disable: Option<bool>,
-	#[serde(rename="port")]
+	#[serde(rename = "port")]
 	_legacy_port: Option<u16>,
-	#[serde(rename="interface")]
+	#[serde(rename = "interface")]
 	_legacy_interface: Option<String>,
-	#[serde(rename="hosts")]
+	#[serde(rename = "hosts")]
 	_legacy_hosts: Option<Vec<String>>,
-	#[serde(rename="cors")]
+	#[serde(rename = "cors")]
 	_legacy_cors: Option<String>,
-	#[serde(rename="path")]
+	#[serde(rename = "path")]
 	_legacy_path: Option<String>,
-	#[serde(rename="user")]
+	#[serde(rename = "user")]
 	_legacy_user: Option<String>,
-	#[serde(rename="pass")]
+	#[serde(rename = "pass")]
 	_legacy_pass: Option<String>,
 }
 
@@ -1676,6 +1681,7 @@ mod tests {
 			// -- API and Console Options
 			// RPC
 			flag_no_jsonrpc: false,
+			flag_jsonrpc_experimental: false,
 			arg_jsonrpc_port: 8545u16,
 			arg_jsonrpc_interface: "local".into(),
 			arg_jsonrpc_cors: "null".into(),
@@ -1958,6 +1964,7 @@ mod tests {
 				server_threads: None,
 				processing_threads: None,
 				max_payload: None,
+				experimental_rpcs: None,
 			}),
 			ipc: Some(Ipc {
 				disable: None,

parity/configuration.rs

@@ -138,6 +138,7 @@ impl Configuration {
 		let compaction = self.args.arg_db_compaction.parse()?;
 		let warp_sync = !self.args.flag_no_warp;
 		let geth_compatibility = self.args.flag_geth;
+		let experimental_rpcs = self.args.flag_jsonrpc_experimental;
 		let ipfs_conf = self.ipfs_config();
 		let secretstore_conf = self.secretstore_config()?;
 		let format = self.format()?;
@@ -377,6 +378,7 @@ impl Configuration {
 				warp_sync: warp_sync,
 				warp_barrier: self.args.arg_warp_barrier,
 				geth_compatibility: geth_compatibility,
+				experimental_rpcs,
 				net_settings: self.network_settings()?,
 				ipfs_conf: ipfs_conf,
 				secretstore_conf: secretstore_conf,
@@ -1418,6 +1420,7 @@ mod tests {
 			compaction: Default::default(),
 			vm_type: Default::default(),
 			geth_compatibility: false,
+			experimental_rpcs: false,
 			net_settings: Default::default(),
 			ipfs_conf: Default::default(),
 			secretstore_conf: Default::default(),

parity/informant.rs

@@ -184,7 +184,7 @@ impl InformantData for LightNodeInformantData {
 	fn executes_transactions(&self) -> bool { false }
 
 	fn is_major_importing(&self) -> bool {
-		self.sync.is_major_importing_no_sync()
+		self.sync.is_major_importing()
 	}
 
 	fn report(&self) -> Report {
@@ -256,16 +256,13 @@ impl<T: InformantData> Informant<T> {
 	}
 
 	pub fn tick(&self) {
-		let elapsed = self.last_tick.read().elapsed();
-		if elapsed < Duration::from_secs(5) {
-			return;
-		}
+		let now = Instant::now();
+		let elapsed = now.duration_since(*self.last_tick.read());
 
 		let (client_report, full_report) = {
 			let mut last_report = self.last_report.lock();
 			let full_report = self.target.report();
 			let diffed = full_report.client_report.clone() - &*last_report;
-			*last_report = full_report.client_report.clone();
 			(diffed, full_report)
 		};
 
@@ -289,7 +286,8 @@ impl<T: InformantData> Informant<T> {
 			return;
 		}
 
-		*self.last_tick.write() = Instant::now();
+		*self.last_tick.write() = now;
+		*self.last_report.lock() = full_report.client_report.clone();
 
 		let paint = |c: Style, t: String| match self.with_color && atty::is(atty::Stream::Stdout) {
 			true => format!("{}", c.paint(t)),
@@ -306,7 +304,7 @@ impl<T: InformantData> Informant<T> {
 							format!("{} blk/s {} tx/s {} Mgas/s",
 								paint(Yellow.bold(), format!("{:7.2}", (client_report.blocks_imported * 1000) as f64 / elapsed.as_milliseconds() as f64)),
 								paint(Yellow.bold(), format!("{:6.1}", (client_report.transactions_applied * 1000) as f64 / elapsed.as_milliseconds() as f64)),
-								paint(Yellow.bold(), format!("{:4}", (client_report.gas_processed / (elapsed.as_milliseconds() * 1000)).low_u64()))
+								paint(Yellow.bold(), format!("{:6.1}", (client_report.gas_processed / 1000).low_u64() as f64 / elapsed.as_milliseconds() as f64))
 							)
 						} else {
 							format!("{} hdr/s",

parity/modules.rs

@@ -14,7 +14,7 @@
 // You should have received a copy of the GNU General Public License
 // along with Parity.  If not, see <http://www.gnu.org/licenses/>.
 
-use std::sync::Arc;
+use std::sync::{Arc, mpsc};
 
 use ethcore::client::BlockChainClient;
 use sync::{self, AttachedProtocol, SyncConfig, NetworkConfiguration, Params, ConnectionFilter};
@@ -25,12 +25,17 @@ pub use sync::{EthSync, SyncProvider, ManageNetwork, PrivateTxHandler};
 pub use ethcore::client::ChainNotify;
 use ethcore_logger::Config as LogConfig;
 
-pub type SyncModules = (Arc<SyncProvider>, Arc<ManageNetwork>, Arc<ChainNotify>);
+pub type SyncModules = (
+	Arc<SyncProvider>,
+	Arc<ManageNetwork>,
+	Arc<ChainNotify>,
+	mpsc::Sender<sync::PriorityTask>,
+);
 
 pub fn sync(
-	sync_cfg: SyncConfig,
-	net_cfg: NetworkConfiguration,
-	client: Arc<BlockChainClient>,
+	config: SyncConfig,
+	network_config: NetworkConfiguration,
+	chain: Arc<BlockChainClient>,
 	snapshot_service: Arc<SnapshotService>,
 	private_tx_handler: Arc<PrivateTxHandler>,
 	provider: Arc<Provider>,
@@ -39,15 +44,20 @@ pub fn sync(
 	connection_filter: Option<Arc<ConnectionFilter>>,
 ) -> Result<SyncModules, sync::Error> {
 	let eth_sync = EthSync::new(Params {
-		config: sync_cfg,
-		chain: client,
-		provider: provider,
-		snapshot_service: snapshot_service,
+		config,
+		chain,
+		provider,
+		snapshot_service,
 		private_tx_handler,
-		network_config: net_cfg,
-		attached_protos: attached_protos,
+		network_config,
+		attached_protos,
 	},
 	connection_filter)?;
 
-	Ok((eth_sync.clone() as Arc<SyncProvider>, eth_sync.clone() as Arc<ManageNetwork>, eth_sync.clone() as Arc<ChainNotify>))
+	Ok((
+		eth_sync.clone() as Arc<SyncProvider>,
+		eth_sync.clone() as Arc<ManageNetwork>,
+		eth_sync.clone() as Arc<ChainNotify>,
+		eth_sync.priority_tasks()
+	))
 }

parity/rpc_apis.rs

@@ -228,6 +228,7 @@ pub struct FullDependencies {
 	pub net_service: Arc<ManageNetwork>,
 	pub updater: Arc<Updater>,
 	pub geth_compatibility: bool,
+	pub experimental_rpcs: bool,
 	pub ws_address: Option<Host>,
 	pub fetch: FetchClient,
 	pub executor: Executor,
@@ -317,7 +318,7 @@ impl FullDependencies {
 					}
 				},
 				Api::Personal => {
-					handler.extend_with(PersonalClient::new(&self.secret_store, dispatcher.clone(), self.geth_compatibility).to_delegate());
+					handler.extend_with(PersonalClient::new(&self.secret_store, dispatcher.clone(), self.geth_compatibility, self.experimental_rpcs).to_delegate());
 				},
 				Api::Signer => {
 					handler.extend_with(SignerClient::new(&self.secret_store, dispatcher.clone(), &self.signer_service, self.executor.clone()).to_delegate());
@@ -438,6 +439,7 @@ pub struct LightDependencies<T> {
 	pub ws_address: Option<Host>,
 	pub fetch: FetchClient,
 	pub geth_compatibility: bool,
+	pub experimental_rpcs: bool,
 	pub executor: Executor,
 	pub whisper_rpc: Option<::whisper::RpcFactory>,
 	pub private_tx_service: Option<Arc<PrivateTransactionManager>>,
@@ -531,7 +533,7 @@ impl<C: LightChainClient + 'static> LightDependencies<C> {
 					handler.extend_with(EthPubSub::to_delegate(client));
 				},
 				Api::Personal => {
-					handler.extend_with(PersonalClient::new(&self.secret_store, dispatcher.clone(), self.geth_compatibility).to_delegate());
+					handler.extend_with(PersonalClient::new(&self.secret_store, dispatcher.clone(), self.geth_compatibility, self.experimental_rpcs).to_delegate());
 				},
 				Api::Signer => {
 					handler.extend_with(SignerClient::new(&self.secret_store, dispatcher.clone(), &self.signer_service, self.executor.clone()).to_delegate());

parity/run.rs

@@ -15,7 +15,7 @@
 // along with Parity.  If not, see <http://www.gnu.org/licenses/>.
 
 use std::any::Any;
-use std::sync::{Arc, Weak};
+use std::sync::{Arc, Weak, atomic};
 use std::time::{Duration, Instant};
 use std::thread;
 
@@ -115,6 +115,7 @@ pub struct RunCmd {
 	pub compaction: DatabaseCompactionProfile,
 	pub vm_type: VMType,
 	pub geth_compatibility: bool,
+	pub experimental_rpcs: bool,
 	pub net_settings: NetworkSettings,
 	pub ipfs_conf: ipfs::Configuration,
 	pub secretstore_conf: secretstore::Configuration,
@@ -312,6 +313,7 @@ fn execute_light_impl(cmd: RunCmd, logger: Arc<RotatingLogger>) -> Result<Runnin
 		ws_address: cmd.ws_conf.address(),
 		fetch: fetch,
 		geth_compatibility: cmd.geth_compatibility,
+		experimental_rpcs: cmd.experimental_rpcs,
 		executor: runtime.executor(),
 		whisper_rpc: whisper_factory,
 		private_tx_service: None, //TODO: add this to client.
@@ -402,11 +404,6 @@ fn execute_impl<Cr, Rr>(cmd: RunCmd, logger: Arc<RotatingLogger>, on_client_rq:
 	// create dirs used by parity
 	cmd.dirs.create_dirs(cmd.acc_conf.unlocked_accounts.len() == 0, cmd.secretstore_conf.enabled)?;
 
-	// run in daemon mode
-	if let Some(pid_file) = cmd.daemon {
-		daemonize(pid_file)?;
-	}
-
 	//print out running parity environment
 	print_running_environment(&spec.data_dir, &cmd.dirs, &db_dirs);
 
@@ -482,7 +479,6 @@ fn execute_impl<Cr, Rr>(cmd: RunCmd, logger: Arc<RotatingLogger>, on_client_rq:
 		cmd.gas_pricer_conf.to_gas_pricer(fetch.clone(), runtime.executor()),
 		&spec,
 		Some(account_provider.clone()),
-
 	));
 	miner.set_author(cmd.miner_extras.author, None).expect("Fails only if password is Some; password is None; qed");
 	miner.set_gas_range_target(cmd.miner_extras.gas_range_target);
@@ -639,7 +635,7 @@ fn execute_impl<Cr, Rr>(cmd: RunCmd, logger: Arc<RotatingLogger>, on_client_rq:
 	};
 
 	// create sync object
-	let (sync_provider, manage_network, chain_notify) = modules::sync(
+	let (sync_provider, manage_network, chain_notify, priority_tasks) = modules::sync(
 		sync_config,
 		net_conf.clone().into(),
 		client.clone(),
@@ -653,6 +649,18 @@ fn execute_impl<Cr, Rr>(cmd: RunCmd, logger: Arc<RotatingLogger>, on_client_rq:
 
 	service.add_notify(chain_notify.clone());
 
+	// Propagate transactions as soon as they are imported.
+	let tx = ::parking_lot::Mutex::new(priority_tasks);
+	let is_ready = Arc::new(atomic::AtomicBool::new(true));
+	miner.add_transactions_listener(Box::new(move |_hashes| {
+		// we want to have only one PendingTransactions task in the queue.
+		if is_ready.compare_and_swap(true, false, atomic::Ordering::SeqCst) {
+			let task = ::sync::PriorityTask::PropagateTransactions(Instant::now(), is_ready.clone());
+			// we ignore error cause it means that we are closing
+			let _ = tx.lock().send(task);
+		}
+	}));
+
 	// provider not added to a notification center is effectively disabled
 	// TODO [debris] refactor it later on
 	if cmd.private_tx_enabled {
@@ -712,6 +720,7 @@ fn execute_impl<Cr, Rr>(cmd: RunCmd, logger: Arc<RotatingLogger>, on_client_rq:
 		net_service: manage_network.clone(),
 		updater: updater.clone(),
 		geth_compatibility: cmd.geth_compatibility,
+		experimental_rpcs: cmd.experimental_rpcs,
 		ws_address: cmd.ws_conf.address(),
 		fetch: fetch.clone(),
 		executor: runtime.executor(),
@@ -737,7 +746,7 @@ fn execute_impl<Cr, Rr>(cmd: RunCmd, logger: Arc<RotatingLogger>, on_client_rq:
 	let secretstore_deps = secretstore::Dependencies {
 		client: client.clone(),
 		sync: sync_provider.clone(),
-		miner: miner,
+		miner: miner.clone(),
 		account_provider: account_provider,
 		accounts_passwords: &passwords,
 	};
@@ -798,6 +807,12 @@ fn execute_impl<Cr, Rr>(cmd: RunCmd, logger: Arc<RotatingLogger>, on_client_rq:
 	client.set_exit_handler(on_client_rq);
 	updater.set_exit_handler(on_updater_rq);
 
+	// run in daemon mode
+	if let Some(pid_file) = cmd.daemon {
+		info!("Running as a daemon process!");
+		daemonize(pid_file)?;
+	}
+
 	Ok(RunningClient {
 		inner: RunningClientInner::Full {
 			rpc: rpc_direct,

rpc/Cargo.toml

@@ -59,6 +59,7 @@ parity-updater = { path = "../updater" }
 parity-version = { path = "../util/version" }
 patricia-trie = "0.3.0"
 rlp = { version = "0.3.0", features = ["ethereum"] }
+eip712 = { path = "../util/EIP-712" }
 stats = { path = "../util/stats" }
 vm = { path = "../ethcore/vm" }
 

rpc/src/lib.rs

@@ -63,6 +63,7 @@ extern crate parity_runtime;
 extern crate parity_updater as updater;
 extern crate parity_version as version;
 extern crate patricia_trie as trie;
+extern crate eip712;
 extern crate rlp;
 extern crate stats;
 extern crate vm;

rpc/src/tests/ws.rs

@@ -28,8 +28,7 @@ use tests::helpers::{GuardedAuthCodes, Server};
 
 /// Setup a mock signer for tests
 pub fn serve() -> (Server<ws::Server>, usize, GuardedAuthCodes) {
-	let port = 35000 + rand::random::<usize>() % 10000;
-	let address = format!("127.0.0.1:{}", port).parse().unwrap();
+	let address = "127.0.0.1:0".parse().unwrap();
 	let io = MetaIoHandler::default();
 	let authcodes = GuardedAuthCodes::new();
 	let stats = Arc::new(informant::RpcStats::default());
@@ -44,6 +43,7 @@ pub fn serve() -> (Server<ws::Server>, usize, GuardedAuthCodes) {
 		extractors::WsExtractor::new(Some(&authcodes.path)),
 		extractors::WsStats::new(stats),
 	).unwrap());
+	let port = res.addr().port() as usize;
 
 	(res, port, authcodes)
 }

rpc/src/v1/extractors.rs

@@ -218,9 +218,10 @@ impl<M: core::Middleware<Metadata>> WsDispatcher<M> {
 
 impl<M: core::Middleware<Metadata>> core::Middleware<Metadata> for WsDispatcher<M> {
 	type Future = Either<
-		core::FutureRpcResult<M::Future>,
+		core::FutureRpcResult<M::Future, M::CallFuture>,
 		core::FutureResponse,
 	>;
+	type CallFuture = core::middleware::NoopCallFuture;
 
 	fn on_request<F, X>(&self, request: core::Request, meta: Metadata, process: F)
 		-> Either<Self::Future, X>

rpc/src/v1/helpers/dispatch.rs

@@ -49,7 +49,8 @@ use v1::types::{
 	RichRawTransaction as RpcRichRawTransaction,
 	ConfirmationPayload as RpcConfirmationPayload,
 	ConfirmationResponse,
-	SignRequest as RpcSignRequest,
+	EthSignRequest as RpcEthSignRequest,
+	EIP191SignRequest as RpcSignRequest,
 	DecryptRequest as RpcDecryptRequest,
 };
 use rlp;
@@ -693,6 +694,19 @@ pub fn execute<D: Dispatcher + 'static>(
 				);
 			Box::new(future::done(res))
 		},
+		ConfirmationPayload::SignMessage(address, data) => {
+			if accounts.is_hardware_address(&address) {
+				return Box::new(future::err(errors::account("Error signing message with hardware_wallet",
+					"Message signing is unsupported")));
+			}
+			let res = signature(&accounts, address, data, pass)
+				.map(|result| result
+					.map(|rsv| H520(rsv.into_electrum()))
+					.map(RpcH520::from)
+					.map(ConfirmationResponse::Signature)
+				);
+			Box::new(future::done(res))
+		},
 		ConfirmationPayload::Decrypt(address, data) => {
 			if accounts.is_hardware_address(&address) {
 				return Box::new(future::err(errors::unsupported("Decrypting via hardware wallets is not supported.", None)));
@@ -775,8 +789,11 @@ pub fn from_rpc<D>(payload: RpcConfirmationPayload, default_account: Address, di
 		RpcConfirmationPayload::Decrypt(RpcDecryptRequest { address, msg }) => {
 			Box::new(future::ok(ConfirmationPayload::Decrypt(address.into(), msg.into())))
 		},
-		RpcConfirmationPayload::EthSignMessage(RpcSignRequest { address, data }) => {
+		RpcConfirmationPayload::EthSignMessage(RpcEthSignRequest { address, data }) => {
 			Box::new(future::ok(ConfirmationPayload::EthSignMessage(address.into(), data.into())))
 		},
+		RpcConfirmationPayload::EIP191SignMessage(RpcSignRequest { address, data }) => {
+			Box::new(future::ok(ConfirmationPayload::SignMessage(address.into(), data.into())))
+		},
 	}
 }

rpc/src/v1/helpers/eip191.rs

@@ -0,0 +1,61 @@
+// Copyright 2015-2018 Parity Technologies (UK) Ltd.
+// This file is part of Parity.
+
+// Parity is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Parity is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
+
+//! EIP-191 compliant decoding + hashing
+use v1::types::{EIP191Version, Bytes, PresignedTransaction};
+use eip712::{hash_structured_data, EIP712};
+use serde_json::{Value, from_value};
+use v1::helpers::errors;
+use jsonrpc_core::Error;
+use v1::helpers::dispatch::eth_data_hash;
+use hash::keccak;
+use std::fmt::Display;
+use ethereum_types::H256;
+
+/// deserializes and hashes the message depending on the version specifier
+pub fn hash_message(version: EIP191Version, message: Value) -> Result<H256, Error> {
+	let data = match version {
+		EIP191Version::StructuredData => {
+			let typed_data = from_value::<EIP712>(message)
+				.map_err(map_serde_err("StructuredData"))?;
+
+			hash_structured_data(typed_data)
+				.map_err(|err| errors::invalid_call_data(err.kind()))?
+		}
+
+		EIP191Version::PresignedTransaction => {
+			let data = from_value::<PresignedTransaction>(message)
+				.map_err(map_serde_err("WithValidator"))?;
+			let prefix = b"\x19\x00";
+			let data = [&prefix[..], &data.validator.0[..], &data.data.0[..]].concat();
+			keccak(data)
+		}
+
+		EIP191Version::PersonalMessage => {
+			let bytes = from_value::<Bytes>(message)
+				.map_err(map_serde_err("Bytes"))?;
+			eth_data_hash(bytes.0)
+		}
+	};
+
+	Ok(data)
+}
+
+fn map_serde_err<T: Display>(struct_name: &'static str) -> impl Fn(T) -> Error {
+	move |error: T| {
+		errors::invalid_call_data(format!("Error deserializing '{}': {}", struct_name, error))
+	}
+}

rpc/src/v1/helpers/errors.rs

@@ -53,6 +53,7 @@ mod codes {
 	pub const FETCH_ERROR: i64 = -32060;
 	pub const NO_LIGHT_PEERS: i64 = -32065;
 	pub const DEPRECATED: i64 = -32070;
+	pub const EXPERIMENTAL_RPC: i64 = -32071;
 }
 
 pub fn unimplemented(details: Option<String>) -> Error {
@@ -286,6 +287,14 @@ pub fn signing(error: AccountError) -> Error {
 	}
 }
 
+pub fn invalid_call_data<T: fmt::Display>(error: T) -> Error {
+	Error {
+		code: ErrorCode::ServerError(codes::ENCODING_ERROR),
+		message: format!("{}", error),
+		data: None
+	}
+}
+
 pub fn password(error: AccountError) -> Error {
 	Error {
 		code: ErrorCode::ServerError(codes::PASSWORD_INVALID),
@@ -492,3 +501,15 @@ pub fn on_demand_others(err: &OnDemandError) -> Error {
 	}
 }
 
+/// Returns a descriptive error in case experimental RPCs are not enabled.
+pub fn require_experimental(allow_experimental_rpcs: bool, eip: &str) -> Result<(), Error> {
+	if allow_experimental_rpcs {
+		Ok(())
+	} else {
+		Err(Error {
+			code: ErrorCode::ServerError(codes::EXPERIMENTAL_RPC),
+			message: format!("This method is not part of the official RPC API yet (EIP-{}). Run with `--jsonrpc-experimental` to enable it.", eip),
+			data: Some(Value::String(format!("See EIP: https://eips.ethereum.org/EIPS/eip-{}", eip))),
+		})
+	}
+}

rpc/src/v1/helpers/light_fetch.rs

@@ -19,12 +19,12 @@
 use std::cmp;
 use std::sync::Arc;
 
-use light::on_demand::error::Error as OnDemandError;
 use ethcore::basic_account::BasicAccount;
 use ethcore::encoded;
 use ethcore::filter::Filter as EthcoreFilter;
 use ethcore::ids::BlockId;
 use ethcore::receipt::Receipt;
+use ethcore::executed::ExecutionError;
 
 use jsonrpc_core::{Result, Error};
 use jsonrpc_core::futures::{future, Future};
@@ -38,6 +38,7 @@ use light::on_demand::{
 	request, OnDemand, HeaderRef, Request as OnDemandRequest,
 	Response as OnDemandResponse, ExecutionResult,
 };
+use light::on_demand::error::Error as OnDemandError;
 use light::request::Field;
 
 use sync::LightSync;
@@ -202,8 +203,8 @@ impl LightFetch {
 	/// Helper for getting proved execution.
 	pub fn proved_read_only_execution(&self, req: CallRequest, num: Trailing<BlockNumber>) -> impl Future<Item = ExecutionResult, Error = Error> + Send {
 		const DEFAULT_GAS_PRICE: u64 = 21_000;
-		// starting gas when gas not provided.
-		const START_GAS: u64 = 50_000;
+		// (21000 G_transaction + 32000 G_create + some marginal to allow a few operations)
+		const START_GAS: u64 = 60_000;
 
 		let (sync, on_demand, client) = (self.sync.clone(), self.on_demand.clone(), self.client.clone());
 		let req: CallRequestHelper = req.into();
@@ -615,28 +616,41 @@ struct ExecuteParams {
 	sync: Arc<LightSync>,
 }
 
-// has a peer execute the transaction with given params. If `gas_known` is false,
-// this will double the gas on each `OutOfGas` error.
+// Has a peer execute the transaction with given params. If `gas_known` is false, this will set the `gas value` to the
+// `required gas value` unless it exceeds the block gas limit
 fn execute_read_only_tx(gas_known: bool, params: ExecuteParams) -> impl Future<Item = ExecutionResult, Error = Error> + Send {
 	if !gas_known {
 		Box::new(future::loop_fn(params, |mut params| {
 			execute_read_only_tx(true, params.clone()).and_then(move |res| {
 				match res {
 					Ok(executed) => {
-						// TODO: how to distinguish between actual OOG and
-						// exception?
-						if executed.exception.is_some() {
-							let old_gas = params.tx.gas;
-							params.tx.gas = params.tx.gas * 2u32;
-							if params.tx.gas > params.hdr.gas_limit() {
-								params.tx.gas = old_gas;
+						// `OutOfGas` exception, try double the gas
+						if let Some(vm::Error::OutOfGas) = executed.exception {
+							// block gas limit already tried, regard as an error and don't retry
+							if params.tx.gas >= params.hdr.gas_limit() {
+								trace!(target: "light_fetch", "OutOutGas exception received, gas increase: failed");
 							} else {
+								params.tx.gas = cmp::min(params.tx.gas * 2_u32, params.hdr.gas_limit());
+								trace!(target: "light_fetch", "OutOutGas exception received, gas increased to {}",
+									   params.tx.gas);
 								return Ok(future::Loop::Continue(params))
 							}
 						}
-
 						Ok(future::Loop::Break(Ok(executed)))
 					}
+					Err(ExecutionError::NotEnoughBaseGas { required, got }) => {
+						trace!(target: "light_fetch", "Not enough start gas provided required: {}, got: {}",
+							   required, got);
+						if required <= params.hdr.gas_limit() {
+							params.tx.gas = required;
+							return Ok(future::Loop::Continue(params))
+						} else {
+							warn!(target: "light_fetch",
+								  "Required gas is bigger than block header's gas dropping the request");
+							Ok(future::Loop::Break(Err(ExecutionError::NotEnoughBaseGas { required, got })))
+						}
+					}
+					// Non-recoverable execution error
 					failed => Ok(future::Loop::Break(failed)),
 				}
 			})

rpc/src/v1/helpers/mod.rs

@@ -25,6 +25,7 @@ pub mod light_fetch;
 pub mod nonce;
 pub mod oneshot;
 pub mod secretstore;
+pub mod eip191;
 
 mod network_settings;
 mod poll_filter;

rpc/src/v1/helpers/requests.rs

@@ -18,6 +18,7 @@ use ethereum_types::{U256, Address};
 use bytes::Bytes;
 
 use v1::types::{Origin, TransactionCondition};
+use ethereum_types::H256;
 
 /// Transaction request coming from RPC
 #[derive(Debug, Clone, Default, Eq, PartialEq, Hash)]
@@ -117,6 +118,8 @@ pub enum ConfirmationPayload {
 	SignTransaction(FilledTransactionRequest),
 	/// Sign a message with an Ethereum specific security prefix.
 	EthSignMessage(Address, Bytes),
+	/// Sign a message
+	SignMessage(Address, H256),
 	/// Decrypt request
 	Decrypt(Address, Bytes),
 }
@@ -127,6 +130,7 @@ impl ConfirmationPayload {
 			ConfirmationPayload::SendTransaction(ref request) => request.from,
 			ConfirmationPayload::SignTransaction(ref request) => request.from,
 			ConfirmationPayload::EthSignMessage(ref address, _) => *address,
+			ConfirmationPayload::SignMessage(ref address, _) => *address,
 			ConfirmationPayload::Decrypt(ref address, _) => *address,
 		}
 	}

rpc/src/v1/impls/parity.rs

@@ -434,7 +434,7 @@ impl<C, M, U, S> Parity for ParityClient<C, M, U> where
 			BlockNumber::Earliest => BlockId::Earliest,
 			BlockNumber::Latest => BlockId::Latest,
 		};
-		let receipts = try_bf!(self.client.block_receipts(id).ok_or_else(errors::unknown_block));
+		let receipts = try_bf!(self.client.localized_block_receipts(id).ok_or_else(errors::unknown_block));
 		Box::new(future::ok(receipts.into_iter().map(Into::into).collect()))
 	}
 

rpc/src/v1/impls/personal.rs

@@ -26,7 +26,7 @@ use ethkey::{public_to_address, recover, Signature};
 
 use jsonrpc_core::{BoxFuture, Result};
 use jsonrpc_core::futures::{future, Future};
-use v1::helpers::errors;
+use v1::helpers::{errors, eip191};
 use v1::helpers::dispatch::{self, eth_data_hash, Dispatcher, SignWith};
 use v1::traits::Personal;
 use v1::types::{
@@ -36,23 +36,33 @@ use v1::types::{
 	ConfirmationResponse as RpcConfirmationResponse,
 	TransactionRequest,
 	RichRawTransaction as RpcRichRawTransaction,
+	EIP191Version,
 };
 use v1::metadata::Metadata;
+use eip712::{EIP712, hash_structured_data};
+use jsonrpc_core::types::Value;
 
 /// Account management (personal) rpc implementation.
 pub struct PersonalClient<D: Dispatcher> {
 	accounts: Arc<AccountProvider>,
 	dispatcher: D,
 	allow_perm_unlock: bool,
+	allow_experimental_rpcs: bool,
 }
 
 impl<D: Dispatcher> PersonalClient<D> {
 	/// Creates new PersonalClient
-	pub fn new(accounts: &Arc<AccountProvider>, dispatcher: D, allow_perm_unlock: bool) -> Self {
+	pub fn new(
+		accounts: &Arc<AccountProvider>,
+		dispatcher: D,
+		allow_perm_unlock: bool,
+		allow_experimental_rpcs: bool,
+	) -> Self {
 		PersonalClient {
 			accounts: accounts.clone(),
 			dispatcher,
 			allow_perm_unlock,
+			allow_experimental_rpcs,
 		}
 	}
 }
@@ -150,6 +160,53 @@ impl<D: Dispatcher + 'static> Personal for PersonalClient<D> {
 				 }))
 	}
 
+	fn sign_191(&self, version: EIP191Version, data: Value, account: RpcH160, password: String) -> BoxFuture<RpcH520> {
+		try_bf!(errors::require_experimental(self.allow_experimental_rpcs, "191"));
+
+		let data = try_bf!(eip191::hash_message(version, data));
+		let dispatcher = self.dispatcher.clone();
+		let accounts = self.accounts.clone();
+
+		let payload = RpcConfirmationPayload::EIP191SignMessage((account.clone(), data.into()).into());
+
+		Box::new(dispatch::from_rpc(payload, account.into(), &dispatcher)
+			.and_then(|payload| {
+				dispatch::execute(dispatcher, accounts, payload, dispatch::SignWith::Password(password.into()))
+			})
+			.map(|v| v.into_value())
+			.then(|res| match res {
+				Ok(RpcConfirmationResponse::Signature(signature)) => Ok(signature),
+				Err(e) => Err(e),
+				e => Err(errors::internal("Unexpected result", e)),
+			})
+		)
+	}
+
+	fn sign_typed_data(&self, typed_data: EIP712, account: RpcH160, password: String) -> BoxFuture<RpcH520> {
+		try_bf!(errors::require_experimental(self.allow_experimental_rpcs, "712"));
+
+		let data = match hash_structured_data(typed_data) {
+			Ok(d) => d,
+			Err(err) => return Box::new(future::err(errors::invalid_call_data(err.kind()))),
+		};
+		let dispatcher = self.dispatcher.clone();
+		let accounts = self.accounts.clone();
+
+		let payload = RpcConfirmationPayload::EIP191SignMessage((account.clone(), data.into()).into());
+
+		Box::new(dispatch::from_rpc(payload, account.into(), &dispatcher)
+			.and_then(|payload| {
+				dispatch::execute(dispatcher, accounts, payload, dispatch::SignWith::Password(password.into()))
+			})
+			.map(|v| v.into_value())
+			.then(|res| match res {
+				Ok(RpcConfirmationResponse::Signature(signature)) => Ok(signature),
+				Err(e) => Err(e),
+				e => Err(errors::internal("Unexpected result", e)),
+			})
+		)
+	}
+
 	fn ec_recover(&self, data: RpcBytes, signature: RpcH520) -> BoxFuture<RpcH160> {
 		let signature: H520 = signature.into();
 		let signature = Signature::from_electrum(&signature);

rpc/src/v1/impls/pubsub.rs

@@ -21,7 +21,7 @@ use std::time::Duration;
 use parking_lot::RwLock;
 
 use jsonrpc_core::{self as core, Result, MetaIoHandler};
-use jsonrpc_core::futures::{Future, Stream, Sink};
+use jsonrpc_core::futures::{future, Future, Stream, Sink};
 use jsonrpc_macros::Trailing;
 use jsonrpc_macros::pubsub::Subscriber;
 use jsonrpc_pubsub::SubscriptionId;
@@ -42,7 +42,7 @@ impl<S: core::Middleware<Metadata>> PubSubClient<S> {
 	/// Creates new `PubSubClient`.
 	pub fn new(rpc: MetaIoHandler<Metadata, S>, executor: Executor) -> Self {
 		let poll_manager = Arc::new(RwLock::new(GenericPollManager::new(rpc)));
-		let pm2 = poll_manager.clone();
+		let pm2 = Arc::downgrade(&poll_manager);
 
 		let timer = tokio_timer::wheel()
 			.tick_duration(Duration::from_millis(500))
@@ -52,7 +52,13 @@ impl<S: core::Middleware<Metadata>> PubSubClient<S> {
 		let interval = timer.interval(Duration::from_millis(1000));
 		executor.spawn(interval
 			.map_err(|e| warn!("Polling timer error: {:?}", e))
-			.for_each(move |_| pm2.read().tick())
+			.for_each(move |_| {
+				if let Some(pm2) = pm2.upgrade() {
+					pm2.read().tick()
+				} else {
+					Box::new(future::err(()))
+				}
+			})
 		);
 
 		PubSubClient {

rpc/src/v1/impls/signer.rs

@@ -215,6 +215,14 @@ impl<D: Dispatcher + 'static> Signer for SignerClient<D> {
 						Err(err) => Err(errors::invalid_params("Invalid signature received.", err)),
 					}
 				},
+				ConfirmationPayload::SignMessage(address, hash) => {
+					let signature = ethkey::Signature::from_electrum(&bytes.0);
+					match ethkey::verify_address(&address, &signature, &hash) {
+						Ok(true) => Ok(ConfirmationResponse::Signature(bytes.0.as_slice().into())),
+						Ok(false) => Err(errors::invalid_params("Sender address does not match the signature.", ())),
+						Err(err) => Err(errors::invalid_params("Invalid signature received.", err)),
+					}
+				},
 				ConfirmationPayload::Decrypt(_address, _data) => {
 					// TODO [ToDr]: Decrypt can we verify if the answer is correct?
 					Ok(ConfirmationResponse::Decrypt(bytes))

rpc/src/v1/informant.rs

@@ -205,6 +205,7 @@ impl<T: ActivityNotifier> Middleware<T> {
 
 impl<M: core::Metadata, T: ActivityNotifier> core::Middleware<M> for Middleware<T> {
 	type Future = core::FutureResponse;
+	type CallFuture = core::middleware::NoopCallFuture;
 
 	fn on_request<F, X>(&self, request: core::Request, meta: M, process: F) -> Either<Self::Future, X> where
 		F: FnOnce(core::Request, M) -> X,

rpc/src/v1/tests/eth.rs

@@ -27,6 +27,7 @@ use ethcore::miner::Miner;
 use ethcore::spec::{Genesis, Spec};
 use ethcore::test_helpers;
 use ethcore::verification::queue::kind::blocks::Unverified;
+use ethcore::verification::VerifierType;
 use ethjson::blockchain::BlockChain;
 use ethjson::spec::ForkSpec;
 use io::IoChannel;
@@ -84,7 +85,16 @@ struct EthTester {
 
 impl EthTester {
 	fn from_chain(chain: &BlockChain) -> Self {
-		let tester = Self::from_spec(make_spec(chain));
+
+		let tester = if ::ethjson::blockchain::Engine::NoProof == chain.engine {
+			let mut config = ClientConfig::default();
+			config.verifier_type = VerifierType::CanonNoSeal;
+			config.check_seal = false;
+			Self::from_spec_conf(make_spec(chain), config)
+		} else {
+			Self::from_spec(make_spec(chain))
+		};
+
 
 		for b in chain.blocks_rlp() {
 			if let Ok(block) = Unverified::from_rlp(b) {
@@ -101,6 +111,12 @@ impl EthTester {
 	}
 
 	fn from_spec(spec: Spec) -> Self {
+		let config = ClientConfig::default();
+		Self::from_spec_conf(spec, config)
+	}
+
+	fn from_spec_conf(spec: Spec, config: ClientConfig) -> Self {
+
 		let runtime = Runtime::with_thread_count(1);
 		let account_provider = account_provider();
 		let opt_account_provider = account_provider.clone();
@@ -108,7 +124,7 @@ impl EthTester {
 		let snapshot_service = snapshot_service();
 
 		let client = Client::new(
-			ClientConfig::default(),
+			config,
 			&spec,
 			test_helpers::new_db(),
 			miner_service.clone(),
@@ -213,9 +229,9 @@ fn eth_get_block_by_hash() {
 	let tester = EthTester::from_chain(&chain);
 
 	// We're looking for block number 4 from "RPC_API_Test_Frontier"
-	let req_block = r#"{"method":"eth_getBlockByHash","params":["0x9c9bdab4cb53fd834e790b13545597f026494d42112e84c0aca9dd6bcc545295",false],"id":1,"jsonrpc":"2.0"}"#;
+	let req_block = r#"{"method":"eth_getBlockByHash","params":["0xaddb9e39795e9e041c936b88a2577802569f34afded0948707b074caa3163a87",false],"id":1,"jsonrpc":"2.0"}"#;
 
-	let res_block = r#"{"jsonrpc":"2.0","result":{"author":"0x8888f1f195afa192cfee860698584c030f4c9db1","difficulty":"0x200c0","extraData":"0x","gasLimit":"0x1dd8112","gasUsed":"0x5458","hash":"0x9c9bdab4cb53fd834e790b13545597f026494d42112e84c0aca9dd6bcc545295","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x8888f1f195afa192cfee860698584c030f4c9db1","mixHash":"0xaddea8d25bb0f955fa6c1d58d74ab8a3fec99d37943e2a261e3b12f97d6bff7c","nonce":"0x8e18bed16d5a88da","number":"0x4","parentHash":"0x2cbf4fc930c5b4c87598f43fc8eb26dccdab2f58a7d0d3ca92ec60a5444a330e","receiptsRoot":"0x7ed8026cf72ed0e98e6fd53ab406e51ffd34397d9da0052494ff41376fda7b5f","sealFields":["0xa0addea8d25bb0f955fa6c1d58d74ab8a3fec99d37943e2a261e3b12f97d6bff7c","0x888e18bed16d5a88da"],"sha3Uncles":"0x75cc08a7cb2cf8081446659fecb2633fb6b922d26edd59bd2272b1f5cae1c78b","size":"0x661","stateRoot":"0x68805721294e365020aca15ed56c360d9dc2cf03cbeff84c9b84b8aed023bfb5","timestamp":"0x59d662ff","totalDifficulty":"0xa0180","transactions":["0xb094b9dc356dbb8b256402c6d5709288066ad6a372c90c9c516f14277545fd58"],"transactionsRoot":"0x97a593d8d7e15b57f5c6bb25bc6c325463ef99f874bc08a78656c3ab5cb23262","uncles":["0xa1e9c9ecd2af999e0723aae1dc55dd9789ca618e0b34badcc8ac7d9a3dad3af2","0x81d429b6b6635214a2b0f976cc4b2ed49808140d6bede50129bc10d22ac9249e"]},"id":1}"#;
+	let res_block = r#"{"jsonrpc":"2.0","result":{"author":"0x8888f1f195afa192cfee860698584c030f4c9db1","difficulty":"0x20080","extraData":"0x","gasLimit":"0x1dd7ea0","gasUsed":"0x5458","hash":"0xaddb9e39795e9e041c936b88a2577802569f34afded0948707b074caa3163a87","logsBloom":"0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000","miner":"0x8888f1f195afa192cfee860698584c030f4c9db1","mixHash":"0x713b0b31f6e72d8cb7367eaf59447ea531f209fc80e6379edd9f8d3bb73931c4","nonce":"0x4534b406bc23b86d","number":"0x4","parentHash":"0x17567aa5995b703736e32972289d68af50543acc4d56d37e8ad1fea7252cac4a","receiptsRoot":"0x7ed8026cf72ed0e98e6fd53ab406e51ffd34397d9da0052494ff41376fda7b5f","sealFields":["0xa0713b0b31f6e72d8cb7367eaf59447ea531f209fc80e6379edd9f8d3bb73931c4","0x884534b406bc23b86d"],"sha3Uncles":"0xe588a44b3e320e72e70b32b531f3ac0d432e756120135ae8fe5fa10895196b40","size":"0x661","stateRoot":"0x68805721294e365020aca15ed56c360d9dc2cf03cbeff84c9b84b8aed023bfb5","timestamp":"0x5bbdf772","totalDifficulty":"0xa00c0","transactions":["0xb094b9dc356dbb8b256402c6d5709288066ad6a372c90c9c516f14277545fd58"],"transactionsRoot":"0x97a593d8d7e15b57f5c6bb25bc6c325463ef99f874bc08a78656c3ab5cb23262","uncles":["0x86b48f5186c4b0882d3dca7977aa37840008832ef092f8ef797019dc74bfa8c7","0x2da9d062c11d536f0f1cc2a4e0111597c79926958d0fc26ae1a2d07d1a3bf47d"]},"id":1}"#;
 	assert_eq!(tester.handler.handle_request_sync(req_block).unwrap(), res_block);
 }
 
@@ -227,6 +243,7 @@ const TRANSACTION_COUNT_SPEC: &'static [u8] = br#"{
 			"params": {
 				"minimumDifficulty": "0x020000",
 				"difficultyBoundDivisor": "0x0800",
+				"blockReward": "0x4563918244F40000",
 				"durationLimit": "0x0d",
 				"homesteadTransition": "0xffffffffffffffff",
 				"daoHardforkTransition": "0xffffffffffffffff",
@@ -237,7 +254,6 @@ const TRANSACTION_COUNT_SPEC: &'static [u8] = br#"{
 	},
 	"params": {
 		"gasLimitBoundDivisor": "0x0400",
-		"blockReward": "0x4563918244F40000",
 		"registrar" : "0xc6d9d2cd449a754c494264e1809c50e34d64562b",
 		"accountStartNonce": "0x00",
 		"maximumExtraDataSize": "0x20",
@@ -276,6 +292,7 @@ const POSITIVE_NONCE_SPEC: &'static [u8] = br#"{
 				"minimumDifficulty": "0x020000",
 				"difficultyBoundDivisor": "0x0800",
 				"durationLimit": "0x0d",
+				"blockReward": "0x4563918244F40000",
 				"homesteadTransition": "0xffffffffffffffff",
 				"daoHardforkTransition": "0xffffffffffffffff",
 				"daoHardforkBeneficiary": "0x0000000000000000000000000000000000000000",
@@ -285,7 +302,6 @@ const POSITIVE_NONCE_SPEC: &'static [u8] = br#"{
 	},
 	"params": {
 		"gasLimitBoundDivisor": "0x0400",
-		"blockReward": "0x4563918244F40000",
 		"registrar" : "0xc6d9d2cd449a754c494264e1809c50e34d64562b",
 		"accountStartNonce": "0x0100",
 		"maximumExtraDataSize": "0x20",

rpc/src/v1/tests/mocked/personal.rs

@@ -25,12 +25,16 @@ use jsonrpc_core::IoHandler;
 use parking_lot::Mutex;
 use transaction::{Action, Transaction};
 use parity_runtime::Runtime;
+use hash::keccak;
 
 use v1::{PersonalClient, Personal, Metadata};
-use v1::helpers::nonce;
+use v1::helpers::{nonce, eip191};
 use v1::helpers::dispatch::{eth_data_hash, FullDispatcher};
 use v1::tests::helpers::TestMinerService;
-use v1::types::H520;
+use v1::types::{EIP191Version, PresignedTransaction, H520};
+use rustc_hex::ToHex;
+use serde_json::to_value;
+use ethkey::Secret;
 
 struct PersonalTester {
 	_runtime: Runtime,
@@ -53,6 +57,16 @@ fn miner_service() -> Arc<TestMinerService> {
 }
 
 fn setup() -> PersonalTester {
+	setup_with(Config {
+		allow_experimental_rpcs: true
+	})
+}
+
+struct Config {
+	pub allow_experimental_rpcs: bool,
+}
+
+fn setup_with(c: Config) -> PersonalTester {
 	let runtime = Runtime::with_thread_count(1);
 	let accounts = accounts_provider();
 	let client = blockchain_client();
@@ -60,7 +74,7 @@ fn setup() -> PersonalTester {
 	let reservations = Arc::new(Mutex::new(nonce::Reservations::new(runtime.executor())));
 
 	let dispatcher = FullDispatcher::new(client, miner.clone(), reservations, 50);
-	let personal = PersonalClient::new(&accounts, dispatcher, false);
+	let personal = PersonalClient::new(&accounts, dispatcher, false, c.allow_experimental_rpcs);
 
 	let mut io = IoHandler::default();
 	io.extend_with(personal.to_delegate());
@@ -328,3 +342,195 @@ fn should_unlock_account_permanently() {
 	assert_eq!(tester.io.handle_request_sync(&request), Some(response.into()));
 	assert!(tester.accounts.sign(address, None, Default::default()).is_ok(), "Should unlock account.");
 }
+
+#[test]
+fn sign_eip191_with_validator() {
+	let tester = setup();
+	let address = tester.accounts.new_account(&"password123".into()).unwrap();
+	let request = r#"{
+		"jsonrpc": "2.0",
+		"method": "personal_sign191",
+		"params": [
+			"0x00",
+			{
+				"validator": ""#.to_owned() + &format!("0x{:x}", address) + r#"",
+				"data": ""# + &format!("0x{:x}", keccak("hello world")) + r#""
+			},
+			""# + &format!("0x{:x}", address) + r#"",
+			"password123"
+		],
+		"id": 1
+	}"#;
+	let with_validator = to_value(PresignedTransaction {
+		validator: address.into(),
+		data: keccak("hello world").to_vec().into()
+	}).unwrap();
+	let result = eip191::hash_message(EIP191Version::PresignedTransaction, with_validator).unwrap();
+	let result = tester.accounts.sign(address, Some("password123".into()), result).unwrap().into_electrum();
+	let expected = r#"{"jsonrpc":"2.0","result":""#.to_owned() +  &format!("0x{}", result.to_hex()) + r#"","id":1}"#;
+	let response = tester.io.handle_request_sync(&request).unwrap();
+	assert_eq!(response, expected)
+}
+
+#[test]
+fn sign_eip191_structured_data() {
+	let tester = setup();
+	let secret: Secret = keccak("cow").into();
+	let address = tester.accounts.insert_account(secret, &"lol".into()).unwrap();
+	let request = r#"{
+		"jsonrpc": "2.0",
+		"method": "personal_sign191",
+		"params": [
+			"0x01",
+			{
+				"primaryType": "Mail",
+				"domain": {
+					"name": "Ether Mail",
+					"version": "1",
+					"chainId": "0x1",
+					"verifyingContract": "0xCcCCccccCCCCcCCCCCCcCcCccCcCCCcCcccccccC"
+				},
+				"message": {
+					"from": {
+						"name": "Cow",
+						"wallet": "0xCD2a3d9F938E13CD947Ec05AbC7FE734Df8DD826"
+					},
+					"to": {
+						"name": "Bob",
+						"wallet": "0xbBbBBBBbbBBBbbbBbbBbbbbBBbBbbbbBbBbbBBbB"
+					},
+					"contents": "Hello, Bob!"
+				},
+				"types": {
+					"EIP712Domain": [
+						{ "name": "name", "type": "string" },
+						{ "name": "version", "type": "string" },
+						{ "name": "chainId", "type": "uint256" },
+						{ "name": "verifyingContract", "type": "address" }
+					],
+					"Person": [
+						{ "name": "name", "type": "string" },
+						{ "name": "wallet", "type": "address" }
+					],
+					"Mail": [
+						{ "name": "from", "type": "Person" },
+						{ "name": "to", "type": "Person" },
+						{ "name": "contents", "type": "string" }
+					]
+				}
+			},
+			""#.to_owned() + &format!("0x{:x}", address) + r#"",
+			"lol"
+		],
+		"id": 1
+	}"#;
+	let expected = r#"{"jsonrpc":"2.0","result":"0x4355c47d63924e8a72e509b65029052eb6c299d53a04e167c5775fd466751c9d07299936d304c153f6443dfa05f40ff007d72911b6f72307f996231605b915621c","id":1}"#;
+	let response = tester.io.handle_request_sync(&request).unwrap();
+	assert_eq!(response, expected)
+}
+
+#[test]
+fn sign_structured_data() {
+	let tester = setup();
+	let secret: Secret = keccak("cow").into();
+	let address = tester.accounts.insert_account(secret, &"lol".into()).unwrap();
+	let request = r#"{
+		"jsonrpc": "2.0",
+		"method": "personal_signTypedData",
+		"params": [
+			{
+				"primaryType": "Mail",
+				"domain": {
+					"name": "Ether Mail",
+					"version": "1",
+					"chainId": "0x1",
+					"verifyingContract": "0xCcCCccccCCCCcCCCCCCcCcCccCcCCCcCcccccccC"
+				},
+				"message": {
+					"from": {
+						"name": "Cow",
+						"wallet": "0xCD2a3d9F938E13CD947Ec05AbC7FE734Df8DD826"
+					},
+					"to": {
+						"name": "Bob",
+						"wallet": "0xbBbBBBBbbBBBbbbBbbBbbbbBBbBbbbbBbBbbBBbB"
+					},
+					"contents": "Hello, Bob!"
+				},
+				"types": {
+					"EIP712Domain": [
+						{ "name": "name", "type": "string" },
+						{ "name": "version", "type": "string" },
+						{ "name": "chainId", "type": "uint256" },
+						{ "name": "verifyingContract", "type": "address" }
+					],
+					"Person": [
+						{ "name": "name", "type": "string" },
+						{ "name": "wallet", "type": "address" }
+					],
+					"Mail": [
+						{ "name": "from", "type": "Person" },
+						{ "name": "to", "type": "Person" },
+						{ "name": "contents", "type": "string" }
+					]
+				}
+			},
+			""#.to_owned() + &format!("0x{:x}", address) + r#"",
+			"lol"
+		],
+		"id": 1
+	}"#;
+	let expected = r#"{"jsonrpc":"2.0","result":"0x4355c47d63924e8a72e509b65029052eb6c299d53a04e167c5775fd466751c9d07299936d304c153f6443dfa05f40ff007d72911b6f72307f996231605b915621c","id":1}"#;
+	let response = tester.io.handle_request_sync(&request).unwrap();
+	assert_eq!(response, expected)
+}
+
+#[test]
+fn should_disable_experimental_apis() {
+	// given
+	let tester = setup_with(Config {
+		allow_experimental_rpcs: false,
+	});
+
+	// when
+	let request = r#"{
+		"jsonrpc": "2.0",
+		"method": "personal_sign191",
+		"params": [
+			"0x01",
+			{},
+			"0x1234567891234567891234567891234567891234",
+			"lol"
+		],
+		"id": 1
+	}"#;
+	let r1 = tester.io.handle_request_sync(&request).unwrap();
+	let request = r#"{
+		"jsonrpc": "2.0",
+		"method": "personal_signTypedData",
+		"params": [
+			{
+				"types": {},
+				"message": {},
+				"domain": {
+					"name": "",
+					"version": "1",
+					"chainId": "0x1",
+					"verifyingContract": "0xCcCCccccCCCCcCCCCCCcCcCccCcCCCcCcccccccC"
+				},
+				"primaryType": ""
+			},
+			"0x1234567891234567891234567891234678912344",
+			"lol"
+		],
+		"id": 1
+	}"#;
+	let r2 = tester.io.handle_request_sync(&request).unwrap();
+
+	// then
+	let expected = r#"{"jsonrpc":"2.0","error":{"code":-32071,"message":"This method is not part of the official RPC API yet (EIP-191). Run with `--jsonrpc-experimental` to enable it.","data":"See EIP: https://eips.ethereum.org/EIPS/eip-191"},"id":1}"#;
+	assert_eq!(r1, expected);
+
+	let expected = r#"{"jsonrpc":"2.0","error":{"code":-32071,"message":"This method is not part of the official RPC API yet (EIP-712). Run with `--jsonrpc-experimental` to enable it.","data":"See EIP: https://eips.ethereum.org/EIPS/eip-712"},"id":1}"#;
+	assert_eq!(r2, expected);
+}

rpc/src/v1/traits/personal.rs

@@ -15,9 +15,10 @@
 // along with Parity.  If not, see <http://www.gnu.org/licenses/>.
 
 //! Personal rpc interface.
+use eip712::EIP712;
+use jsonrpc_core::types::Value;
 use jsonrpc_core::{BoxFuture, Result};
-
-use v1::types::{Bytes, U128, H160, H256, H520, TransactionRequest, RichRawTransaction as RpcRichRawTransaction};
+use v1::types::{Bytes, U128, H160, H256, H520, TransactionRequest, RichRawTransaction as RpcRichRawTransaction, EIP191Version};
 
 build_rpc_trait! {
 	/// Personal rpc interface. Safe (read-only) functions.
@@ -42,6 +43,15 @@ build_rpc_trait! {
 		#[rpc(name = "personal_sign")]
 		fn sign(&self, Bytes, H160, String) -> BoxFuture<H520>;
 
+		/// Produces an EIP-712 compliant signature with given account using the given password to unlock the
+		/// account during the request.
+		#[rpc(name = "personal_signTypedData")]
+		fn sign_typed_data(&self, EIP712, H160, String) -> BoxFuture<H520>;
+
+		/// Signs an arbitrary message based on the version specified
+		#[rpc(name = "personal_sign191")]
+		fn sign_191(&self, EIP191Version, Value, H160, String) -> BoxFuture<H520>;
+
 		/// Returns the account associated with the private key that was used to calculate the signature in
 		/// `personal_sign`.
 		#[rpc(name = "personal_ecRecover")]

rpc/src/v1/types/confirmations.rs

@@ -59,31 +59,62 @@ impl fmt::Display for ConfirmationPayload {
 			ConfirmationPayload::SendTransaction(ref transaction) => write!(f, "{}", transaction),
 			ConfirmationPayload::SignTransaction(ref transaction) => write!(f, "(Sign only) {}", transaction),
 			ConfirmationPayload::EthSignMessage(ref sign) => write!(f, "{}", sign),
+			ConfirmationPayload::EIP191SignMessage(ref sign) => write!(f, "{}", sign),
 			ConfirmationPayload::Decrypt(ref decrypt) => write!(f, "{}", decrypt),
 		}
 	}
 }
 
-/// Sign request
+/// Ethereum-prefixed Sign request
 #[derive(Debug, Clone, Eq, PartialEq, Hash, Serialize, Deserialize)]
 #[serde(deny_unknown_fields)]
-pub struct SignRequest {
+pub struct EthSignRequest {
 	/// Address
 	pub address: H160,
 	/// Hash to sign
 	pub data: Bytes,
 }
 
-impl From<(H160, Bytes)> for SignRequest {
-	fn from(tuple: (H160, Bytes)) -> Self {
-		SignRequest {
+/// EIP191 Sign request
+#[derive(Debug, Clone, Eq, PartialEq, Hash, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct EIP191SignRequest {
+	/// Address
+	pub address: H160,
+	/// Hash to sign
+	pub data: H256,
+}
+
+impl From<(H160, H256)> for EIP191SignRequest {
+	fn from(tuple: (H160, H256)) -> Self {
+		EIP191SignRequest {
 			address: tuple.0,
 			data: tuple.1,
 		}
 	}
 }
 
-impl fmt::Display for SignRequest {
+impl fmt::Display for EIP191SignRequest {
+	fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+		write!(
+			f,
+			"sign 0x{} with {}",
+			self.data.0.pretty(),
+			Colour::White.bold().paint(format!("0x{:?}", self.address)),
+		)
+	}
+}
+
+impl From<(H160, Bytes)> for EthSignRequest {
+	fn from(tuple: (H160, Bytes)) -> Self {
+		EthSignRequest {
+			address: tuple.0,
+			data: tuple.1,
+		}
+	}
+}
+
+impl fmt::Display for EthSignRequest {
 	fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
 		write!(
 			f,
@@ -169,8 +200,10 @@ pub enum ConfirmationPayload {
 	#[serde(rename="signTransaction")]
 	SignTransaction(TransactionRequest),
 	/// Signature
-	#[serde(rename="sign")]
-	EthSignMessage(SignRequest),
+	#[serde(rename = "sign")]
+	EthSignMessage(EthSignRequest),
+	/// signature without prefix
+	EIP191SignMessage(EIP191SignRequest),
 	/// Decryption
 	#[serde(rename="decrypt")]
 	Decrypt(DecryptRequest),
@@ -181,7 +214,11 @@ impl From<helpers::ConfirmationPayload> for ConfirmationPayload {
 		match c {
 			helpers::ConfirmationPayload::SendTransaction(t) => ConfirmationPayload::SendTransaction(t.into()),
 			helpers::ConfirmationPayload::SignTransaction(t) => ConfirmationPayload::SignTransaction(t.into()),
-			helpers::ConfirmationPayload::EthSignMessage(address, data) => ConfirmationPayload::EthSignMessage(SignRequest {
+			helpers::ConfirmationPayload::EthSignMessage(address, data) => ConfirmationPayload::EthSignMessage(EthSignRequest {
+				address: address.into(),
+				data: data.into(),
+			}),
+			helpers::ConfirmationPayload::SignMessage(address, data) => ConfirmationPayload::EIP191SignMessage(EIP191SignRequest {
 				address: address.into(),
 				data: data.into(),
 			}),

rpc/src/v1/types/eip191.rs

@@ -0,0 +1,57 @@
+// Copyright 2015-2018 Parity Technologies (UK) Ltd.
+// This file is part of Parity.
+
+// Parity is free software: you can redistribute it and/or modify
+// it under the terms of the GNU General Public License as published by
+// the Free Software Foundation, either version 3 of the License, or
+// (at your option) any later version.
+
+// Parity is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+// GNU General Public License for more details.
+
+// You should have received a copy of the GNU General Public License
+// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
+
+//! EIP-191 specific types
+use serde::{Deserialize, Deserializer};
+use serde::de;
+use v1::types::{H160, Bytes};
+
+/// EIP-191 version specifier
+#[derive(Debug)]
+pub enum EIP191Version {
+	/// byte specifier for structured data (0x01)
+	StructuredData,
+	/// byte specifier for personal message (0x45)
+	PersonalMessage,
+	/// byte specifier for presignedtransaction (0x00)
+	PresignedTransaction
+}
+
+/// EIP-191 version 0x0 struct
+#[derive(Deserialize, Serialize)]
+#[serde(rename_all = "camelCase")]
+pub struct PresignedTransaction {
+	// address of intended validator
+	pub validator: H160,
+	// application specific data
+	pub data: Bytes
+}
+
+impl<'de> Deserialize<'de> for EIP191Version {
+	fn deserialize<D>(deserializer: D) -> Result<Self, D::Error>
+		where
+			D: Deserializer<'de>,
+	{
+		let s = String::deserialize(deserializer)?;
+		let byte_version = match s.as_str() {
+			"0x00" => EIP191Version::PresignedTransaction,
+			"0x01" => EIP191Version::StructuredData,
+			"0x45" => EIP191Version::PersonalMessage,
+			other => return Err(de::Error::custom(format!("Invalid byte version '{}'", other))),
+		};
+		Ok(byte_version)
+	}
+}

rpc/src/v1/types/mod.rs

@@ -43,9 +43,10 @@ mod transaction_condition;
 mod uint;
 mod work;
 mod private_receipt;
+mod eip191;
 
 pub mod pubsub;
-
+pub use self::eip191::{EIP191Version, PresignedTransaction};
 pub use self::account_info::{AccountInfo, ExtAccountInfo, HwAccountInfo};
 pub use self::bytes::Bytes;
 pub use self::block::{RichBlock, Block, BlockTransactions, Header, RichHeader, Rich};
@@ -53,7 +54,7 @@ pub use self::block_number::{BlockNumber, LightBlockNumber, block_number_to_id};
 pub use self::call_request::CallRequest;
 pub use self::confirmations::{
 	ConfirmationPayload, ConfirmationRequest, ConfirmationResponse, ConfirmationResponseWithToken,
-	TransactionModification, SignRequest, DecryptRequest, Either
+	TransactionModification, EIP191SignRequest, EthSignRequest, DecryptRequest, Either
 };
 pub use self::consensus_status::*;
 pub use self::derivation::{DeriveHash, DeriveHierarchical, Derive};

rust-toolchain

@@ -1 +0,0 @@
-stable

scripts/docker/hub/Dockerfile

@@ -18,7 +18,6 @@ RUN rm -rf /tmp/* /var/tmp/* /var/lib/apt/lists/*
 RUN groupadd -g 1000 parity \
   && useradd -m -u 1000 -g parity -s /bin/sh parity
 
-USER parity
 
 WORKDIR /home/parity
 
@@ -31,7 +30,10 @@ COPY artifacts/x86_64-unknown-linux-gnu/$TARGET ./bin/$TARGET
 RUN echo "#!/bin/bash \n ${TARGET} \$@" > ./entrypoint.sh
 RUN chmod +x ./entrypoint.sh
 
-ADD check_sync.sh /check_sync.sh
+COPY scripts/docker/hub/check_sync.sh /check_sync.sh
+
+# switch to user parity here
+USER parity
 
 # setup ENTRYPOINT
 EXPOSE 5001 8080 8082 8083 8545 8546 8180 30303/tcp 30303/udp

scripts/gitlab/build-unix.sh

@@ -6,18 +6,23 @@ set -u # treat unset variables as error
 echo "__________Show ENVIROMENT__________"
 echo "CI_SERVER_NAME:   " $CI_SERVER_NAME
 echo "CARGO_HOME:       " $CARGO_HOME
-echo "BUILD_TARGET:     " $BUILD_TARGET
-echo "BUILD_ARCH:       " $BUILD_ARCH
 echo "CARGO_TARGET:     " $CARGO_TARGET
 echo "CC:               " $CC
 echo "CXX:              " $CXX
 
 echo "__________CARGO CONFIG__________"
-mkdir -p .cargo
-rm -f .cargo/config
-echo "[target.$CARGO_TARGET]" >> .cargo/config
-echo "linker= \"$CC\"" >> .cargo/config
-cat .cargo/config
+if [ "${CARGO_TARGET}" = "armv7-linux-androideabi" ]
+then
+  # use build container's cargo config
+  cat /.cargo/config
+else
+  mkdir -p .cargo
+  rm -f .cargo/config
+  echo "[target.$CARGO_TARGET]" >> .cargo/config
+  echo "linker= \"$CC\"" >> .cargo/config
+  cat .cargo/config
+fi
+
 
 echo "_____ Building target: "$CARGO_TARGET" _____"
 time cargo build --target $CARGO_TARGET --release --features final
@@ -32,15 +37,31 @@ mkdir -p artifacts
 cd artifacts
 mkdir -p $CARGO_TARGET
 cd $CARGO_TARGET
-cp ../../target/$CARGO_TARGET/release/parity ./parity
-cp ../../target/$CARGO_TARGET/release/parity-evm ./parity-evm
-cp ../../target/$CARGO_TARGET/release/ethstore ./ethstore
-cp ../../target/$CARGO_TARGET/release/ethkey ./ethkey
-cp ../../target/$CARGO_TARGET/release/whisper ./whisper
-strip -v ./*
+cp -v ../../target/$CARGO_TARGET/release/parity ./parity
+cp -v ../../target/$CARGO_TARGET/release/parity-evm ./parity-evm
+cp -v ../../target/$CARGO_TARGET/release/ethstore ./ethstore
+cp -v ../../target/$CARGO_TARGET/release/ethkey ./ethkey
+cp -v ../../target/$CARGO_TARGET/release/whisper ./whisper
+
+
+# stripping can also be done on release build time
+# export RUSTFLAGS="${RUSTFLAGS} -C link-arg=-s"
+if [ "${CARGO_TARGET}" = "armv7-linux-androideabi" ]
+then
+  arm-linux-androideabi-strip -v ./*
+else
+  strip -v ./*
+fi
+
 echo "_____ Calculating checksums _____"
 for binary in $(ls)
 do
   rhash --sha256 $binary -o $binary.sha256
-  ./parity tools hash $binary > $binary.sha3
+  if [ "${CARGO_TARGET}" = "armv7-linux-androideabi" ]
+  then
+    echo "> ${binary} cannot be hashed with cross-compiled binary"
+  else
+    ./parity tools hash $binary > $binary.sha3
+  fi
 done
+

scripts/gitlab/build-windows.sh

@@ -8,8 +8,6 @@ set LIB="C:\vs2015\VC\lib;C:\Program Files (x86)\Windows Kits\10\Lib\10.0.10240.
 echo "__________Show ENVIROMENT__________"
 echo "CI_SERVER_NAME:   " $CI_SERVER_NAME
 echo "CARGO_HOME:       " $CARGO_HOME
-echo "BUILD_TARGET:     " $BUILD_TARGET
-echo "BUILD_ARCH:       " $BUILD_ARCH
 echo "CARGO_TARGET:     " $CARGO_TARGET
 
 echo "_____ Building target: "$CARGO_TARGET" _____"

scripts/gitlab/cargo-audit.sh

@@ -3,5 +3,5 @@
 set -e # fail on any error
 set -u # treat unset variables as error
 
-CARGO_TARGET_DIR=./target cargo +stable install cargo-audit --force
-cargo +stable audit
+cargo install cargo-audit
+cargo audit

util/EIP-712/Cargo.toml

@@ -0,0 +1,23 @@
+[package]
+name = "eip712"
+version = "0.1.0"
+authors = ["Parity Technologies <admin@parity.io>"]
+
+[dependencies]
+serde_derive = "1.0"
+serde = "1.0"
+serde_json = "1.0"
+ethabi = "6.0"
+keccak-hash = "0.1"
+ethereum-types = "0.4"
+failure = "0.1"
+itertools = "0.7"
+failure_derive = "0.1"
+lazy_static = "1.1"
+toolshed = "0.4"
+regex = "1.0"
+validator = "0.8"
+validator_derive = "0.8"
+lunarity-lexer = "0.1"
+rustc-hex = "2.0"
+indexmap = "1.0.2"