beta backports for 2.0.3 (#9229)

* parity-version: bump beta to 2.0.2

* remove ssl from dockerfiles, closes #8880 (#9195)

* snap: remove ssl dependencies from snapcraft definition (#9222)

* parity-version: bump beta to 2.0.3

* Remove all dapp permissions related settings (#9120)

* Completely remove all dapps struct from rpc

* Remove unused pub use

* Remove dapp policy/permission func in ethcore

* Remove all dapps settings from rpc

* Fix rpc tests

* Use both origin and user_agent

* Address grumbles

* Address grumbles

* Fix tests

* Check if synced when using eth_getWork (#9193) (#9210)

* Check if synced when using eth_getWork (#9193)

* Don't use fn syncing

* Fix identation

* Fix typo

* Don't check for warping

* rpc: avoid calling queue_info twice on eth_getWork

* Fix potential as_usize overflow when casting from U256 in miner (#9221)

* Allow old blocks from peers with lower difficulty (#9226)

Previously we only allow downloading of old blocks if the peer
difficulty was greater than our syncing difficulty. This change allows
downloading of blocks from peers where the difficulty is greater then
the last downloaded old block.

* Update Dockerfile (#9242)

* Update Dockerfile

fix Docker build

* fix dockerfile paths: parity -> parity-ethereum (#9248)

* Propagate transactions for next 4 blocks. (#9265)

Closes #9255 

This PR also removes the limit of max 64 transactions per packet, currently we only attempt to prevent the packet size to go over 8MB. This will only be the case for super-large transactions or high-block-gas-limit chains.

Patching this is important only for chains that have blocks that can fit more than 4k transactions (over 86M block gas limit)

For mainnet, we should actually see a tiny bit faster propagation since instead of computing 4k pending set, we only need `4 * 8M / 21k = 1523` transactions.

Running some tests on `dekompile` node right now, to check how it performs in the wild.

* Update tobalaba.json (#9313)

* Fix load share (#9321)

* fix(light_sync): calculate `load_share` properly

* refactor(api.rs): extract `light_params` fn, add test

* style(api.rs): add trailing commas

* ethcore: fix pow difficulty validation (#9328)

* ethcore: fix pow difficulty validation

* ethcore: validate difficulty is not zero

* ethcore: add issue link to regression test

* ethcore: fix tests

* ethcore: move difficulty_to_boundary to ethash crate

* ethcore: reuse difficulty_to_boundary and boundary_to_difficulty

* ethcore: fix grumbles in difficulty_to_boundary_aux

* Light client `Provide default nonce in transactions when it´s missing` (#9370)

* Provide `default_nonce` in tx`s when it´s missing

When `nonce` is missing in a `EthTransaction` will cause it to fall in
these cases provide `default_nonce` value instead!

* Changed http:// to https:// on Yasm link (#9369)

Changed http:// to https:// on Yasm link in README.md

* Provide `default_nonce` in tx`s when it´s missing

When `nonce` is missing in a `EthTransaction` will cause it to fall in
these cases provide `default_nonce` value instead!

* Address grumbles

* ethcore: kovan: delay activation of strict score validation (#9406)

* Better support for eth_getLogs in light mode (#9186)

* Light client on-demand request for headers range.

* Cache headers in HeaderWithAncestors response.

Also fulfills request locally if all headers are in cache.

* LightFetch::logs fetches missing headers on demand.

* LightFetch::logs limit the number of headers requested at a time.

* LightFetch::logs refactor header fetching logic.

* Enforce limit on header range length in light client logs request.

* Fix light request tests after struct change.

* Respond to review comments.

* Add update docs script to CI (#9219)

* Add update docs script to CI

Added a script to CI that will use the jsonrpc tool to update rpc
documentation then commit and push those to the wiki repo.

* fix gitlab ci lint

* Only apply jsonrpc docs update on tags

* Update gitlab-rpc-docs.sh

* Copy correct parity repo to jsonrpc folder

Copy correct parity repo to jsonrpc folder before attempting to build docs since the CI runner clones the repo as parity and not parity-ethereum.

* Fix JSONRPC docs CI job

Update remote config in wiki repo before pushing changes using a github
token for authentication. Add message to wiki tag when pushing changes.
Use project directory to correctly copy parity code base into the
jsonrpc repo for doc generation.

* Fix set_remote_wiki function call in CI

* Prevent blockchain & miner racing when accessing pending block. (#9310)

* Prevent blockchain & miner racing when accessing pending block.

* Fix unavailability of pending block during reseal.

* Prevent sync restart if import queue full (#9381)

* Add POA Networks: Core and Sokol (#9413)

* ethcore: add poa network and sokol chainspecs

* rpc: simplify chain spec docs

* cli: rearrange networks by main/test and size/range

* parity: don't blacklist 0x00a328 on sokol testnet

* parity: add sokol and poanet to params and clean up a bit, add tests

* ethcore: add the poa networks and clean up a bit

* ethcore: fix path to poacore chain spec

* parity: rename poa networks to poacore and poasokol

* parity: fix configuration tests

* parity: fix parameter tests

* ethcore: rename POA Core and POA Sokol

* Update tobalaba.json (#9419)

* Update hardcoded sync (#9421)

- Update foundation hardcoded header to block 6219777 
- Update ropsten hardcoded header to block 3917825 
- Update kovan hardcoded header to block 8511489

.gitignore

@@ -38,6 +38,7 @@ node_modules
 
 # Build artifacts
 out/
+parity-clib-examples/cpp/build/
 
 .vscode
 rls/

.gitlab-ci.yml

@@ -2,18 +2,18 @@ stages:
   - test
   - push-release
   - build
+  - docs
 variables:
   RUST_BACKTRACE: "1"
   RUSTFLAGS: ""
   CARGOFLAGS: ""
   CI_SERVER_NAME: "GitLab CI"
-  LIBSSL: "libssl1.0.0 (>=1.0.0)"
 cache:
   key: "$CI_BUILD_STAGE-$CI_BUILD_REF_NAME"
   paths:
     - target/
   untracked: true
-linux-ubuntu:
+linux-amd64:
   stage: build
   image: parity/rust:gitlab-ci
   only:
@@ -24,46 +24,13 @@ linux-ubuntu:
   script:
     - rustup default stable
     # ARGUMENTS: 1. BUILD_PLATFORM (target for binaries) 2. PLATFORM (target for cargo) 3. ARC (architecture) 4. & 5. CC & CXX flags 6. binary identifier
-    - scripts/gitlab-build.sh x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu amd64 gcc g++ ubuntu
+    - scripts/gitlab-build.sh x86_64-unknown-linux-gnu x86_64-unknown-linux-gnu amd64 gcc g++ linux
   tags:
     - rust-stable
   artifacts:
     paths:
     - parity.zip
     name: "stable-x86_64-unknown-linux-gnu_parity"
-linux-debian:
-  stage: build
-  image: parity/rust-debian:gitlab-ci
-  only:
-    - beta
-    - tags
-    - stable
-    - triggers
-  script:
-    - export LIBSSL="libssl1.1 (>=1.1.0)"
-    - scripts/gitlab-build.sh x86_64-unknown-debian-gnu x86_64-unknown-linux-gnu amd64 gcc g++ debian
-  tags:
-    - rust-debian
-  artifacts:
-    paths:
-    - parity.zip
-    name: "stable-x86_64-unknown-debian-gnu_parity"
-linux-centos:
-  stage: build
-  image: parity/rust-centos:gitlab-ci
-  only:
-    - beta
-    - tags
-    - stable
-    - triggers
-  script:
-    - scripts/gitlab-build.sh x86_64-unknown-centos-gnu x86_64-unknown-linux-gnu x86_64 gcc g++ centos
-  tags:
-    - rust-centos
-  artifacts:
-    paths:
-    - parity.zip
-    name: "x86_64-unknown-centos-gnu_parity"
 linux-i686:
   stage: build
   image: parity/rust-i686:gitlab-ci
@@ -73,7 +40,7 @@ linux-i686:
     - stable
     - triggers
   script:
-    - scripts/gitlab-build.sh i686-unknown-linux-gnu i686-unknown-linux-gnu i386 gcc g++ ubuntu
+    - scripts/gitlab-build.sh i686-unknown-linux-gnu i686-unknown-linux-gnu i386 gcc g++ linux
   tags:
     - rust-i686
   artifacts:
@@ -90,7 +57,7 @@ linux-armv7:
     - stable
     - triggers
   script:
-    - scripts/gitlab-build.sh armv7-unknown-linux-gnueabihf armv7-unknown-linux-gnueabihf armhf arm-linux-gnueabihf-gcc arm-linux-gnueabihf-g++ ubuntu
+    - scripts/gitlab-build.sh armv7-unknown-linux-gnueabihf armv7-unknown-linux-gnueabihf armhf arm-linux-gnueabihf-gcc arm-linux-gnueabihf-g++ linux
   tags:
     - rust-arm
   artifacts:
@@ -98,7 +65,7 @@ linux-armv7:
     - parity.zip
     name: "armv7_unknown_linux_gnueabihf_parity"
   allow_failure: true
-linux-arm:
+linux-armhf:
   stage: build
   image: parity/rust-arm:gitlab-ci
   only:
@@ -107,7 +74,7 @@ linux-arm:
     - stable
     - triggers
   script:
-    - scripts/gitlab-build.sh arm-unknown-linux-gnueabihf arm-unknown-linux-gnueabihf armhf arm-linux-gnueabihf-gcc arm-linux-gnueabihf-g++ ubuntu
+    - scripts/gitlab-build.sh arm-unknown-linux-gnueabihf arm-unknown-linux-gnueabihf armhf arm-linux-gnueabihf-gcc arm-linux-gnueabihf-g++ linux
   tags:
     - rust-arm
   artifacts:
@@ -124,7 +91,7 @@ linux-aarch64:
     - stable
     - triggers
   script:
-    - scripts/gitlab-build.sh aarch64-unknown-linux-gnu aarch64-unknown-linux-gnu arm64 aarch64-linux-gnu-gcc aarch64-linux-gnu-g++ ubuntu
+    - scripts/gitlab-build.sh aarch64-unknown-linux-gnu aarch64-unknown-linux-gnu arm64 aarch64-linux-gnu-gcc aarch64-linux-gnu-g++ linux
   tags:
     - rust-arm
   artifacts:
@@ -201,6 +168,7 @@ docker-build:
   stage: build
   only:
     - tags
+    - master
     - beta
     - stable
     - triggers
@@ -253,6 +221,15 @@ test-rust-nightly:
     - rust
     - rust-nightly
   allow_failure: true
+json-rpc-docs:
+  stage: docs
+  only:
+    - tags
+  image: parity/rust:gitlab-ci
+  script:
+    - scripts/gitlab-rpc-docs.sh
+  tags:
+    - docs
 push-release:
   stage: push-release
   only:

CHANGELOG.md

@@ -1,3 +1,193 @@
+## Parity [v1.11.6](https://github.com/paritytech/parity/releases/tag/v1.11.6) (2018-07-09)
+
+Parity 1.11.6 is a bug-fix release to improve performance and stability.
+
+The full list of included changes:
+
+- Beta: 1.11.6 backports ([#9015](https://github.com/paritytech/parity/pull/9015))
+  - Parity-version: bump beta to 1.11.6
+  - Scripts: remove md5 checksums ([#8884](https://github.com/paritytech/parity/pull/8884))
+  - Add support for --chain tobalaba
+    - Convert indents to tabs :)
+  - Fixes for misbehavior reporting in AuthorityRound ([#8998](https://github.com/paritytech/parity/pull/8998))
+    - Aura: only report after checking for repeated skipped primaries
+    - Aura: refactor duplicate code for getting epoch validator set
+    - Aura: verify_external: report on validator set contract instance
+    - Aura: use correct validator set epoch number when reporting
+    - Aura: use epoch set when verifying blocks
+    - Aura: report skipped primaries when generating seal
+    - Aura: handle immediate transitions
+    - Aura: don't report skipped steps from genesis to first block
+    - Aura: fix reporting test
+    - Aura: refactor duplicate code to handle immediate_transitions
+    - Aura: let reporting fail on verify_block_basic
+    - Aura: add comment about possible failure of reporting
+  - Only return error log for rustls ([#9025](https://github.com/paritytech/parity/pull/9025))
+  - Transaction Pool improvements ([#8470](https://github.com/paritytech/parity/pull/8470))
+    - Don't use ethereum_types in transaction pool.
+    - Hide internal insertion_id.
+    - Fix tests.
+    - Review grumbles.
+  - Improve should_replace on NonceAndGasPrice ([#8980](https://github.com/paritytech/parity/pull/8980))
+    - Additional tests for NonceAndGasPrice::should_replace.
+    - Fix should_replace in the distinct sender case.
+    - Use natural priority ordering to simplify should_replace.
+  - Minimal effective gas price in the queue ([#8934](https://github.com/paritytech/parity/pull/8934))
+    - Minimal effective gas price.
+    - Fix naming, add test
+    - Fix minimal entry score and add test.
+    - Fix worst_transaction.
+    - Remove effective gas price threshold.
+    - Don't leak gas_price decisions out of Scoring.
+  - Never drop local transactions from different senders. ([#9002](https://github.com/paritytech/parity/pull/9002))
+  - Recently rejected cache for transaction queue ([#9005](https://github.com/paritytech/parity/pull/9005))
+    - Store recently rejected transactions.
+    - Don't cache AlreadyImported rejections.
+    - Make the size of transaction verification queue dependent on pool size.
+    - Add a test for recently rejected.
+    - Fix logging for recently rejected.
+    - Make rejection cache smaller.
+    - Obsolete test removed
+    - Obsolete test removed
+    - Construct cache with_capacity.
+  - Optimize pending transactions filter ([#9026](https://github.com/paritytech/parity/pull/9026))
+    - Rpc: return unordered transactions in pending transactions filter
+    - Ethcore: use LruCache for nonce cache
+      - Only clear the nonce cache when a block is retracted
+    - Revert "ethcore: use LruCache for nonce cache"
+      - This reverts commit b382c19.
+    - Use only cached nonces when computing pending hashes.
+    - Give filters their own locks, so that they don't block one another.
+    - Fix pending transaction count if not sealing.
+    - Clear cache only when block is enacted.
+    - Fix RPC tests.
+    - Address review comments.
+  - A last bunch of txqueue performance optimizations ([#9024](https://github.com/paritytech/parity/pull/9024))
+    - Clear cache only when block is enacted.
+    - Add tracing for cull.
+    - Cull split.
+    - Cull after creating pending block.
+    - Add constant, remove sync::read tracing.
+    - Reset debug.
+    - Remove excessive tracing.
+    - Use struct for NonceCache.
+    - Fix build
+    - Remove warnings.
+    - Fix build again.
+  - Miner: add missing macro use for trace_time
+  - Ci: remove md5 merge leftovers
+
+## Parity [v1.11.5](https://github.com/paritytech/parity/releases/tag/v1.11.5) (2018-06-29)
+
+Parity 1.11.5 is a bug-fix release to improve performance and stability.
+
+The full list of included changes:
+
+- Bump beta to 1.11.5 / Backports ([#8955](https://github.com/paritytech/parity/pull/8955))
+  - Parity-version: bump beta to 1.11.5
+  - Update ropsten.json ([#8926](https://github.com/paritytech/parity/pull/8926))
+  - Update hardcoded headers ([#8925](https://github.com/paritytech/parity/pull/8925))
+    - Update kovan.json
+      - Update Kovan to block 7693549
+    - Update foundation.json
+      - Updated to block 5812225
+    - Update ropsten.json
+      - Update to 3465217
+  - Scripts: minor improvements ([#8930](https://github.com/paritytech/parity/pull/8930))
+    - CI: enable 'latest' docker tag on master pipeline
+    - CI: mark both beta and stable as stable snap.
+    - CI: sign all windows binaries
+  - Scripts: fix docker build tag on latest using master ([#8952](https://github.com/paritytech/parity/pull/8952))
+  - Rpc: cap gas limit of local calls ([#8943](https://github.com/paritytech/parity/pull/8943))
+  - Snap: downgrade rust to revision 1.26.2, ref snapcraft/+bug/1778530 ([#8984](https://github.com/paritytech/parity/pull/8984))
+    - Snap: downgrade rust to revision 1.26.2, ref snapcraft/+bug/1778530
+    - Snap: use plugin rust
+  - Fix deadlock in blockchain. ([#8977](https://github.com/paritytech/parity/pull/8977))
+  - Remove js-glue from workspace
+    - This fixes test error on Rust 1.27 but also prevents js-glue from building itself.
+    - Builtin dapp users can still use js-glue from crates.io.
+  - Fix Android build on beta ([#9003](https://github.com/paritytech/parity/pull/9003))
+
+## Parity [v1.11.4](https://github.com/paritytech/parity/releases/tag/v1.11.4) (2018-06-20)
+
+Parity 1.11.4 is a bug-fix release to improve performance and stability.
+
+The full list of included changes:
+
+- Backports ([#8916](https://github.com/paritytech/parity/pull/8916))
+  - `Duration_ns: u64 -> duration: Duration` ([#8457](https://github.com/paritytech/parity/pull/8457))
+    - Duration_ns: u64 -> duration: Duration
+    - Format on millis {:.2} -> {}
+  - Keep all enacted blocks notify in order ([#8524](https://github.com/paritytech/parity/pull/8524))
+    - Keep all enacted blocks notify in order
+    - Collect is unnecessary
+    - Update ChainNotify to use ChainRouteType
+    - Fix all ethcore fn defs
+    - Wrap the type within ChainRoute
+    - Fix private-tx and sync api
+    - Fix secret_store API
+    - Fix updater API
+    - Fix rpc api
+    - Fix informant api
+    - Eagerly cache enacted/retracted and remove contain_enacted/retracted
+    - Fix indent
+    - Tests: should use full expr form for struct constructor
+    - Use into_enacted_retracted to further avoid copy
+    - Typo: not a function
+    - Rpc/tests: ChainRoute -> ChainRoute::new
+  - Handle removed logs in filter changes and add geth compatibility field ([#8796](https://github.com/paritytech/parity/pull/8796))
+    - Add removed geth compatibility field in log
+    - Fix mocked tests
+    - Add field block hash in PollFilter
+    - Store last block hash info for log filters
+    - Implement canon route
+    - Use canon logs for fetching reorg logs
+    - Make sure removed flag is set
+    - Address grumbles
+  - Fixed AuthorityRound deadlock on shutdown, closes [#8088](https://github.com/paritytech/parity/issues/8088) ([#8803](https://github.com/paritytech/parity/pull/8803))
+  - Ci: Fix docker tags ([#8822](https://github.com/paritytech/parity/pull/8822))
+    - Scripts: enable docker builds for beta and stable
+    - Scripts: docker latest should be beta not master
+    - Scripts: docker latest is master
+  - Ethcore: fix ancient block error msg handling ([#8832](https://github.com/paritytech/parity/pull/8832))
+  - Disable parallel verification and skip verifiying already imported txs. ([#8834](https://github.com/paritytech/parity/pull/8834))
+    - Reject transactions that are already in pool without verifying them.
+    - Avoid verifying already imported transactions.
+  - Fix concurrent access to signer queue ([#8854](https://github.com/paritytech/parity/pull/8854))
+    - Fix concurrent access to signer queue
+    - Put request back to the queue if confirmation failed
+    - Typo: fix docs and rename functions to be more specific
+    - Change trace info "Transaction" -> "Request"
+  - Don't allocate in expect_valid_rlp unless necessary ([#8867](https://github.com/paritytech/parity/pull/8867))
+    - Don't allocate via format! in case there's no error
+    - Fix test?
+  - Fixed ipc leak, closes [#8774](https://github.com/paritytech/parity/issues/8774) ([#8876](https://github.com/paritytech/parity/pull/8876))
+  - Add new ovh bootnodes and fix port for foundation bootnode 3.2 ([#8886](https://github.com/paritytech/parity/pull/8886))
+    - Add new ovh bootnodes and fix port for foundation bootnode 3.2
+    - Remove old bootnodes.
+    - Remove duplicate 1118980bf48b0a3640bdba04e0fe78b1add18e1cd99bf22d53daac1fd9972ad650df52176e7c7d89d1114cfef2bc23a2959aa54998a46afcf7d91809f0855082
+  - Block 0 is valid in queries ([#8891](https://github.com/paritytech/parity/pull/8891))
+  - Add ETC Cooperative-run load balanced parity node ([#8892](https://github.com/paritytech/parity/pull/8892))
+  - Minor fix in chain supplier and light provider ([#8906](https://github.com/paritytech/parity/pull/8906))
+    - Fix chain supplier increment
+    - Fix light provider block_headers
+  - Check whether we need resealing in miner and unwrap has_account in account_provider ([#8853](https://github.com/paritytech/parity/pull/8853))
+    - Remove unused Result wrap in has_account
+    - Check whether we need to reseal for external transactions
+    - Fix reference to has_account interface
+    - Typo: missing )
+    - Refactor duplicates to prepare_and_update_sealing
+    - Fix build
+  - Allow disabling local-by-default for transactions with new config entry ([#8882](https://github.com/paritytech/parity/pull/8882))
+    - Add tx_queue_allow_unknown_local config option
+    - Refactor flag name + don't change import_own_tx behaviour
+    - Add fn to TestMinerService
+    - Avoid race condition from trusted sources
+- Parity-version: beta release 1.11.4 ([#8856](https://github.com/paritytech/parity/pull/8856))
+  - Cherry-pick network-specific release flag ([#8821](https://github.com/paritytech/parity/pull/8821))
+  - Parity-version: bump beta to 1.11.4
+  - Parity-version: remove merge leftovers
+
 ## Parity [v1.11.3](https://github.com/paritytech/parity/releases/tag/v1.11.3) (2018-06-06)
 
 Parity 1.11.3 is a security-relevant release. Please upgrade your nodes as soon as possible to [v1.10.6](https://github.com/paritytech/parity/releases/tag/v1.10.6) or [v1.11.3](https://github.com/paritytech/parity/releases/tag/v1.11.3).

Cargo.toml

@@ -1,12 +1,13 @@
 [package]
 description = "Parity Ethereum client"
-name = "parity"
+name = "parity-ethereum"
 # NOTE Make sure to update util/version/Cargo.toml as well
-version = "1.12.0"
+version = "2.0.3"
 license = "GPL-3.0"
 authors = ["Parity Technologies <admin@parity.io>"]
 
 [dependencies]
+blooms-db = { path = "util/blooms-db" }
 log = "0.3"
 env_logger = "0.4"
 rustc-hex = "1.0"
@@ -19,7 +20,7 @@ number_prefix = "0.2"
 rpassword = "1.0"
 semver = "0.9"
 ansi_term = "0.10"
-parking_lot = "0.5"
+parking_lot = "0.6"
 regex = "0.2"
 atty = "0.2.8"
 toml = "0.4"
@@ -31,8 +32,8 @@ futures-cpupool = "0.1"
 fdlimit = "0.1"
 ctrlc = { git = "https://github.com/paritytech/rust-ctrlc.git" }
 jsonrpc-core = { git = "https://github.com/paritytech/jsonrpc.git", branch = "parity-1.11" }
-ethcore = { path = "ethcore" }
-ethcore-bytes = { path = "util/bytes" }
+ethcore = { path = "ethcore", features = ["parity"] }
+parity-bytes = { git = "https://github.com/paritytech/parity-common" }
 ethcore-io = { path = "util/io" }
 ethcore-light = { path = "ethcore/light" }
 ethcore-logger = { path = "logger" }
@@ -45,8 +46,7 @@ ethcore-transaction = { path = "ethcore/transaction" }
 ethereum-types = "0.3"
 node-filter = { path = "ethcore/node_filter" }
 ethkey = { path = "ethkey" }
-node-health = { path = "dapps/node-health" }
-rlp = { path = "util/rlp" }
+rlp = { git = "https://github.com/paritytech/parity-common" }
 rpc-cli = { path = "rpc_cli" }
 parity-hash-fetch = { path = "hash-fetch" }
 parity-ipfs-api = { path = "ipfs" }
@@ -57,17 +57,16 @@ parity-rpc-client = { path = "rpc_client" }
 parity-updater = { path = "updater" }
 parity-version = { path = "util/version" }
 parity-whisper = { path = "whisper" }
-path = { path = "util/path" }
+path = { git = "https://github.com/paritytech/parity-common" }
 dir = { path = "util/dir" }
 panic_hook = { path = "util/panic_hook" }
-keccak-hash = { path = "util/hash" }
+keccak-hash = { git = "https://github.com/paritytech/parity-common" }
 migration-rocksdb = { path = "util/migration-rocksdb" }
-kvdb = { path = "util/kvdb" }
-kvdb-rocksdb = { path = "util/kvdb-rocksdb" }
+kvdb = { git = "https://github.com/paritytech/parity-common" }
+kvdb-rocksdb = { git = "https://github.com/paritytech/parity-common" }
 journaldb = { path = "util/journaldb" }
 mem = { path = "util/mem" }
 
-parity-dapps = { path = "dapps", optional = true }
 ethcore-secretstore = { path = "secret_store", optional = true }
 
 registrar = { path = "registrar" }
@@ -88,8 +87,7 @@ winapi = { version = "0.3.4", features = ["winsock2", "winuser", "shellapi"] }
 daemonize = { git = "https://github.com/paritytech/daemonize" }
 
 [features]
-default = ["dapps"]
-dapps = ["parity-dapps"]
+miner-debug = ["ethcore/miner-debug"]
 json-tests = ["ethcore/json-tests"]
 test-heavy = ["ethcore/test-heavy"]
 evm-debug = ["ethcore/evm-debug"]
@@ -105,6 +103,10 @@ deadlock_detection = ["parking_lot/deadlock_detection"]
 # `valgrind --tool=massif /path/to/parity <parity params>`
 # and `massif-visualizer` for visualization
 memory_profiling = []
+# hardcode version number 1.3.7 of parity to force an update
+# in order to manually test that parity fall-over to the local version
+# in case of invalid or deprecated command line arguments are entered
+test-updater = ["parity-updater/test-updater"]
 
 [lib]
 path = "parity/lib.rs"
@@ -114,17 +116,13 @@ path = "parity/main.rs"
 name = "parity"
 
 [profile.dev]
-panic = "abort"
 
 [profile.release]
 debug = false
-lto = false
-panic = "abort"
 
 [workspace]
 members = [
 	"chainspec",
-	"dapps/js-glue",
 	"ethcore/wasm/run",
 	"ethcore/types",
 	"ethkey/cli",
@@ -135,6 +133,9 @@ members = [
 	"transaction-pool",
 	"whisper",
 	"whisper/cli",
+	"util/triehash-ethereum",
+	"util/keccak-hasher",
+	"util/patricia-trie-ethereum",
 ]
 
 [patch.crates-io]

README.md

@@ -48,7 +48,7 @@ We recommend installing Rust through [rustup](https://www.rustup.rs/). If you do
 	$ curl https://sh.rustup.rs -sSf | sh
 	```
 
-	Parity also requires `gcc`, `g++`, `libssl-dev`/`openssl`, `libudev-dev` and `pkg-config` packages to be installed.
+	Parity also requires `gcc`, `g++`, `libssl-dev`/`openssl`, `libudev-dev`, `pkg-config`, `file` and `make` packages to be installed.
 
 - OSX:
 	```bash

dapps/Cargo.toml

@@ -1,43 +0,0 @@
-[package]
-description = "Parity Dapps crate"
-name = "parity-dapps"
-version = "1.12.0"
-license = "GPL-3.0"
-authors = ["Parity Technologies <admin@parity.io>"]
-
-[lib]
-
-[dependencies]
-base32 = "0.3"
-futures = "0.1"
-futures-cpupool = "0.1"
-linked-hash-map = "0.5"
-log = "0.3"
-parity-dapps-glue = "1.9"
-parking_lot = "0.5"
-mime_guess = "2.0.0-alpha.2"
-rand = "0.4"
-rustc-hex = "1.0"
-serde = "1.0"
-serde_derive = "1.0"
-serde_json = "1.0"
-unicase = "1.4"
-zip = { version = "0.3", default-features = false, features = ["deflate"] }
-itertools = "0.5"
-
-jsonrpc-core = { git = "https://github.com/paritytech/jsonrpc.git", branch = "parity-1.11" }
-jsonrpc-http-server = { git = "https://github.com/paritytech/jsonrpc.git", branch = "parity-1.11" }
-
-ethcore-bytes = { path = "../util/bytes" }
-ethereum-types = "0.3"
-fetch = { path = "../util/fetch" }
-node-health = { path = "./node-health" }
-parity-hash-fetch = { path = "../hash-fetch" }
-parity-reactor = { path = "../util/reactor" }
-keccak-hash = { path = "../util/hash" }
-parity-version = { path = "../util/version" }
-registrar = { path = "../registrar" }
-
-[dev-dependencies]
-env_logger = "0.4"
-ethcore-devtools = { path = "../devtools" }

dapps/js-glue/Cargo.toml

@@ -1,28 +0,0 @@
-[package]
-description = "Base Package for all Parity built-in dapps"
-name = "parity-dapps-glue"
-version = "1.9.1"
-license = "GPL-3.0"
-authors = ["Parity Technologies <admin@parity.io>"]
-build = "build.rs"
-
-[build-dependencies]
-quasi_codegen = { version = "0.32", optional = true }
-syntex = { version = "0.58", optional = true }
-
-[dependencies]
-glob = { version = "0.2.11" }
-mime_guess = { version = "2.0.0-alpha.2" }
-aster = { version = "0.41", default-features = false }
-quasi = { version = "0.32", default-features = false }
-quasi_macros = { version = "0.32", optional = true }
-syntex = { version = "0.58", optional = true }
-syntex_syntax = { version = "0.58", optional = true }
-
-[features]
-default = ["with-syntex"]
-nightly = ["quasi_macros"]
-with-syntex = ["quasi/with-syntex", "quasi_codegen", "quasi_codegen/with-syntex", "syntex", "syntex_syntax"]
-use-precompiled-js = []
-
-

dapps/js-glue/README.md

@@ -1,65 +0,0 @@
-# Parity Dapps (JS-glue)
-
-Code generator to simplify creating a built-in Parity Dapp
-
-# How to create new builtin Dapp.
-1. Clone this repository.
-
-   ```bash
-   $ git clone https://github.com/paritytech/parity.git
-   ```
-
-1. Create a new directory for your Dapp. (`./myapp`)
-
-   ```bash
-   $ mkdir -p ./parity/dapps/myapp/src/web
-   ```
-
-1. Copy your frontend files to `./dapps/myapp/src/web` (bundled ones)
-
-   ```bash
-   $ cp -r ./myapp-src/* ./parity/dapps/myapp/src/web
-   ```
-
-1. Instead of creating `web3` in your app. Load (as the first script tag in `head`):
-
-   ```html
-   <script src="/parity-utils/inject.js"></script>
-   ```
-
-   The `inject.js` script will create global `web3` instance with proper provider that should be used by your dapp.
-
-1. Create `./parity/dapps/myapp/Cargo.toml` with you apps details. See example here: [parity-status Cargo.toml](https://github.com/paritytech/parity-ui/blob/master/status/Cargo.toml).
-
-   ```bash
-   $ git clone https://github.com/paritytech/parity-ui.git
-   $ cd ./parity-ui/
-   $ cp ./home/Cargo.toml ../parity/dapps/myapp/Cargo.toml
-   $ cp ./home/build.rs ../parity/dapps/myapp/build.rs
-   $ cp ./home/src/lib.rs ../parity/dapps/myapp/src/lib.rs
-   $ cp ./home/src/lib.rs.in ../parity/dapps/myapp/src/lib.rs.in
-   # And edit the details of your app
-   $ vim ../parity/dapps/myapp/Cargo.toml # Edit the details
-   $ vim ./parity/dapps/myapp/src/lib.rs.in # Edit the details
-   ```
-# How to include your Dapp into `Parity`?
-1. Edit `dapps/Cargo.toml` and add dependency to your application (it can be optional)
-
-   ```toml
-   # Use git repo and version
-   parity-dapps-myapp = { path="./myapp" }
-   ```
-
-1. Edit `dapps/src/apps.rs` and add your application to `all_pages` (if it's optional you need to specify two functions - see `parity-dapps-wallet` example)
-
-1. Compile parity.
-
-   ```bash
-   $ cargo build --release # While inside `parity`
-   ```
-
-1. Commit the results.
-
-   ```bash
-   $ git add myapp && git commit -am "My first Parity Dapp".
-   ```

dapps/js-glue/build.rs

@@ -1,42 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-#[cfg(feature = "with-syntex")]
-mod inner {
-	extern crate syntex;
-	extern crate quasi_codegen;
-
-	use std::env;
-	use std::path::Path;
-
-	pub fn main() {
-		let out_dir = env::var_os("OUT_DIR").unwrap();
-
-		let src = Path::new("src/lib.rs.in");
-		let dst = Path::new(&out_dir).join("lib.rs");
-
-		quasi_codegen::expand(&src, &dst).unwrap();
-	}
-}
-
-#[cfg(not(feature = "with-syntex"))]
-mod inner {
-	pub fn main() {}
-}
-
-fn main() {
-	inner::main();
-}

dapps/js-glue/src/build.rs

@@ -1,80 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-#[cfg(feature = "with-syntex")]
-pub mod inner {
-	use syntex;
-	use codegen;
-	use syntax::{ast, fold};
-	use std::env;
-	use std::path::Path;
-
-	fn strip_attributes(krate: ast::Crate) -> ast::Crate {
-		/// Helper folder that strips the serde attributes after the extensions have been expanded.
-		struct StripAttributeFolder;
-
-		impl fold::Folder for StripAttributeFolder {
-			fn fold_attribute(&mut self, attr: ast::Attribute) -> Option<ast::Attribute> {
-				if &*attr.value.name.as_str() == "webapp" {
-					return None;
-				}
-
-				Some(attr)
-			}
-
-			fn fold_mac(&mut self, mac: ast::Mac) -> ast::Mac {
-				fold::noop_fold_mac(mac, self)
-			}
-		}
-
-		fold::Folder::fold_crate(&mut StripAttributeFolder, krate)
-	}
-
-	pub fn register(reg: &mut syntex::Registry) {
-		reg.add_attr("feature(custom_derive)");
-		reg.add_attr("feature(custom_attribute)");
-
-		reg.add_decorator("derive_WebAppFiles", codegen::expand_webapp_implementation);
-		reg.add_post_expansion_pass(strip_attributes);
-	}
-
-	pub fn generate() {
-		let out_dir = env::var_os("OUT_DIR").unwrap();
-		let mut registry = syntex::Registry::new();
-		register(&mut registry);
-
-		let src = Path::new("src/lib.rs.in");
-		let dst = Path::new(&out_dir).join("lib.rs");
-
-		registry.expand("", &src, &dst).unwrap();
-	}
-}
-
-#[cfg(not(feature = "with-syntex"))]
-pub mod inner {
-	use codegen;
-
-	pub fn register(reg: &mut rustc_plugin::Registry) {
-		reg.register_syntax_extension(
-			syntax::parse::token::intern("derive_WebAppFiles"),
-			syntax::ext::base::MultiDecorator(
-				Box::new(codegen::expand_webapp_implementation)));
-
-		reg.register_attribute("webapp".to_owned(), AttributeType::Normal);
-	}
-
-	pub fn generate() {}
-}

dapps/js-glue/src/codegen.rs

@@ -1,194 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-extern crate aster;
-extern crate glob;
-extern crate mime_guess;
-
-use self::mime_guess::guess_mime_type;
-use std::path::{self, Path, PathBuf};
-use std::ops::Deref;
-
-use syntax::attr;
-use syntax::ast::{self, MetaItem, Item};
-use syntax::codemap::Span;
-use syntax::ext::base::{Annotatable, ExtCtxt};
-use syntax::print::pprust::lit_to_string;
-use syntax::symbol::InternedString;
-
-pub fn expand_webapp_implementation(
-	cx: &mut ExtCtxt,
-	span: Span,
-	meta_item: &MetaItem,
-	annotatable: &Annotatable,
-	push: &mut FnMut(Annotatable)
-) {
-	let item = match *annotatable {
-		Annotatable::Item(ref item) => item,
-		_ => {
-			cx.span_err(meta_item.span, "`#[derive(WebAppFiles)]` may only be applied to struct implementations");
-			return;
-		},
-	};
-	let builder = aster::AstBuilder::new().span(span);
-	implement_webapp(cx, &builder, item, push);
-}
-
-fn implement_webapp(cx: &ExtCtxt, builder: &aster::AstBuilder, item: &Item, push: &mut FnMut(Annotatable)) {
-	let static_files_dir = extract_path(cx, item);
-
-	let src = Path::new("src");
-	let static_files = {
-		let mut buf = src.to_path_buf();
-		buf.push(static_files_dir.deref());
-		buf
-	};
-
-	let search_location = {
-		let mut buf = static_files.to_path_buf();
-		buf.push("**");
-		buf.push("*");
-		buf
-	};
-
-	let files = glob::glob(search_location.to_str().expect("Valid UTF8 path"))
-		.expect("The sources directory is missing.")
-		.collect::<Result<Vec<PathBuf>, glob::GlobError>>()
-		.expect("There should be no error when reading a list of files.");
-
-	let statements = files
-		.iter()
-		.filter(|path_buf| path_buf.is_file())
-		.map(|path_buf| {
-			let path = path_buf.as_path();
-			let filename = path.file_name().and_then(|s| s.to_str()).expect("Only UTF8 paths.");
-			let mime_type = guess_mime_type(filename).to_string();
-			let file_path = as_uri(path.strip_prefix(&static_files).ok().expect("Prefix is always there, cause it's absolute path;qed"));
-			let file_path_in_source = path.to_str().expect("Only UTF8 paths.");
-
-			let path_lit = builder.expr().str(file_path.as_str());
-			let mime_lit = builder.expr().str(mime_type.as_str());
-			let web_path_lit = builder.expr().str(file_path_in_source);
-			let separator_lit = builder.expr().str(path::MAIN_SEPARATOR.to_string().as_str());
-			let concat_id = builder.id("concat!");
-			let env_id = builder.id("env!");
-			let macro_id = builder.id("include_bytes!");
-
-			let content = quote_expr!(
-				cx,
-				$macro_id($concat_id($env_id("CARGO_MANIFEST_DIR"), $separator_lit, $web_path_lit))
-			);
-			quote_stmt!(
-				cx,
-				files.insert($path_lit, File { path: $path_lit, content_type: $mime_lit, content: $content });
-			).expect("The statement is always ok, because it just uses literals.")
-		}).collect::<Vec<ast::Stmt>>();
-
-	let type_name = item.ident;
-
-	let files_impl = quote_item!(cx,
-		impl $type_name {
-			#[allow(unused_mut)]
-			fn files() -> ::std::collections::HashMap<&'static str, File> {
-				let mut files = ::std::collections::HashMap::new();
-				$statements
-				files
-			}
-		}
-	).unwrap();
-
-	push(Annotatable::Item(files_impl));
-}
-
-fn extract_path(cx: &ExtCtxt, item: &Item) -> String {
-	for meta_items in item.attrs.iter().filter_map(webapp_meta_items) {
-		for meta_item in meta_items {
-			let is_path = &*meta_item.name.as_str() == "path";
-			match meta_item.node {
-				ast::MetaItemKind::NameValue(ref lit) if is_path => {
-					if let Some(s) = get_str_from_lit(cx, lit) {
-						return s.deref().to_owned();
-					}
-				},
-				_ => {},
-			}
-		}
-	}
-
-	// default
-	"web".to_owned()
-}
-
-fn webapp_meta_items(attr: &ast::Attribute) -> Option<Vec<ast::MetaItem>> {
-	let is_webapp = &*attr.value.name.as_str() == "webapp";
-	match attr.value.node {
-		ast::MetaItemKind::List(ref items) if is_webapp => {
-			attr::mark_used(&attr);
-			Some(
-				items.iter()
-				.map(|item| item.node.clone())
-				.filter_map(|item| match item {
-					ast::NestedMetaItemKind::MetaItem(item) => Some(item),
-					_ => None,
-				})
-				.collect()
-			)
-		}
-		_ => None
-	}
-}
-
-fn get_str_from_lit(cx: &ExtCtxt, lit: &ast::Lit) -> Option<InternedString> {
-	match lit.node {
-		ast::LitKind::Str(ref s, _) => Some(s.clone().as_str()),
-		_ => {
-			cx.span_err(
-				lit.span,
-				&format!("webapp annotation path must be a string, not `{}`",
-					lit_to_string(lit)
-				)
-			);
-			return None;
-		}
-	}
-}
-
-fn as_uri(path: &Path) -> String {
-	let mut s = String::new();
-	for component in path.iter() {
-		s.push_str(component.to_str().expect("Only UTF-8 filenames are supported."));
-		s.push('/');
-	}
-	s[0..s.len()-1].into()
-}
-
-#[test]
-fn should_convert_path_separators_on_all_platforms() {
-	// given
-	let p = {
-		let mut p = PathBuf::new();
-		p.push("web");
-		p.push("src");
-		p.push("index.html");
-		p
-	};
-
-	// when
-	let path = as_uri(&p);
-
-	// then
-	assert_eq!(path, "web/src/index.html".to_owned());
-}

dapps/js-glue/src/js.rs

@@ -1,89 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-#![cfg_attr(feature = "use-precompiled-js", allow(dead_code))]
-#![cfg_attr(feature = "use-precompiled-js", allow(unused_imports))]
-
-use std::fmt;
-use std::process::Command;
-
-#[cfg(not(windows))]
-mod platform {
-	use std::process::Command;
-
-	pub static NPM_CMD: &'static str = "npm";
-	pub fn handle_cmd(cmd: &mut Command) -> &mut Command {
-		cmd
-	}
-}
-
-#[cfg(windows)]
-mod platform {
-	use std::process::{Command, Stdio};
-
-	pub static NPM_CMD: &'static str = "cmd.exe";
-	// NOTE [ToDr] For some reason on windows
-	// The command doesn't have %~dp0 set properly
-	// and it cannot load globally installed node.exe
-	pub fn handle_cmd(cmd: &mut Command) -> &mut Command {
-		cmd.stdin(Stdio::null())
-			.arg("/c")
-			.arg("npm.cmd")
-	}
-}
-
-fn die<T : fmt::Debug>(s: &'static str, e: T) -> ! {
-	panic!("Error: {}: {:?}", s, e);
-}
-
-#[cfg(feature = "use-precompiled-js")]
-pub fn test(_path: &str) {
-}
-#[cfg(feature = "use-precompiled-js")]
-pub fn build(_path: &str, _dest: &str) {
-}
-
-#[cfg(not(feature = "use-precompiled-js"))]
-pub fn build(path: &str, dest: &str) {
-	let child = platform::handle_cmd(&mut Command::new(platform::NPM_CMD))
-		.arg("install")
-		.arg("--no-progress")
-		.current_dir(path)
-		.status()
-		.unwrap_or_else(|e| die("Installing node.js dependencies with npm", e));
-	assert!(child.success(), "There was an error installing dependencies.");
-
-	let child = platform::handle_cmd(&mut Command::new(platform::NPM_CMD))
-		.arg("run")
-		.arg("build")
-		.env("NODE_ENV", "production")
-		.env("BUILD_DEST", dest)
-		.current_dir(path)
-		.status()
-		.unwrap_or_else(|e| die("Building JS code", e));
-	assert!(child.success(), "There was an error build JS code.");
-}
-
-#[cfg(not(feature = "use-precompiled-js"))]
-pub fn test(path: &str) {
-	let child = Command::new(platform::NPM_CMD)
-		.arg("run")
-		.arg("test")
-		.current_dir(path)
-		.status()
-		.unwrap_or_else(|e| die("Running test command", e));
-	assert!(child.success(), "There was an error while running JS tests.");
-}

dapps/js-glue/src/lib.rs

@@ -1,37 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-#![cfg_attr(not(feature = "with-syntex"), feature(rustc_private, plugin))]
-#![cfg_attr(not(feature = "with-syntex"), plugin(quasi_macros))]
-
-#[cfg(feature = "with-syntex")]
-extern crate syntex;
-
-#[cfg(feature = "with-syntex")]
-extern crate syntex_syntax as syntax;
-
-#[cfg(feature = "with-syntex")]
-include!(concat!(env!("OUT_DIR"), "/lib.rs"));
-
-#[cfg(not(feature = "with-syntex"))]
-#[macro_use]
-extern crate syntax;
-
-#[cfg(not(feature = "with-syntex"))]
-extern crate rustc_plugin;
-
-#[cfg(not(feature = "with-syntex"))]
-include!("lib.rs.in");

dapps/js-glue/src/lib.rs.in

@@ -1,47 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity. If not, see <http://www.gnu.org/licenses/>.
-
-
-extern crate quasi;
-
-mod codegen;
-mod build;
-pub mod js;
-pub use build::inner::generate;
-
-use std::default::Default;
-
-#[derive(Clone)]
-pub struct File {
-	pub path: &'static str,
-	pub content: &'static [u8],
-	// TODO: use strongly-typed MIME.
-	pub content_type: &'static str,
-}
-
-#[derive(Clone, Debug)]
-pub struct Info {
-	pub name: &'static str,
-	pub version: &'static str,
-	pub author: &'static str,
-	pub description: &'static str,
-	pub icon_url: &'static str,
-}
-
-pub trait WebApp : Default + Send + Sync {
-	fn file(&self, path: &str) -> Option<&File>;
-	fn info(&self) -> Info;
-}

dapps/node-health/Cargo.toml

@@ -1,18 +0,0 @@
-[package]
-name = "node-health"
-description = "Node's health status"
-version = "0.1.0"
-license = "GPL-3.0"
-authors = ["Parity Technologies <admin@parity.io>"]
-
-[dependencies]
-futures = "0.1"
-futures-cpupool = "0.1"
-log = "0.3"
-ntp = "0.3.0"
-parking_lot = "0.5"
-serde = "1.0"
-serde_derive = "1.0"
-time = "0.1.35"
-
-parity-reactor = { path = "../../util/reactor" }

dapps/node-health/src/health.rs

@@ -1,122 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Reporting node's health.
-
-use std::sync::Arc;
-use std::time::Duration;
-use futures::Future;
-use futures::sync::oneshot;
-use types::{HealthInfo, HealthStatus, Health};
-use time::{TimeChecker, MAX_DRIFT};
-use parity_reactor::Remote;
-use parking_lot::Mutex;
-use {SyncStatus};
-
-const TIMEOUT: Duration = Duration::from_secs(5);
-const PROOF: &str = "Only one closure is invoked.";
-
-/// A struct enabling you to query for node's health.
-#[derive(Debug, Clone)]
-pub struct NodeHealth {
-	sync_status: Arc<SyncStatus>,
-	time: TimeChecker,
-	remote: Remote,
-}
-
-impl NodeHealth {
-	/// Creates new `NodeHealth`.
-	pub fn new(sync_status: Arc<SyncStatus>, time: TimeChecker, remote: Remote) -> Self {
-		NodeHealth { sync_status, time, remote, }
-	}
-
-	/// Query latest health report.
-	pub fn health(&self) -> Box<Future<Item = Health, Error = ()> + Send> {
-		trace!(target: "dapps", "Checking node health.");
-		// Check timediff
-		let sync_status = self.sync_status.clone();
-		let time = self.time.time_drift();
-		let (tx, rx) = oneshot::channel();
-		let tx = Arc::new(Mutex::new(Some(tx)));
-		let tx2 = tx.clone();
-		self.remote.spawn_with_timeout(
-			move |_| time.then(move |result| {
-				let _ = tx.lock().take().expect(PROOF).send(Ok(result));
-				Ok(())
-			}),
-			TIMEOUT,
-			move || {
-				let _ = tx2.lock().take().expect(PROOF).send(Err(()));
-			},
-		);
-
-		Box::new(rx.map_err(|err| {
-			warn!(target: "dapps", "Health request cancelled: {:?}", err);
-		}).and_then(move |time| {
-			// Check peers
-			let peers = {
-				let (connected, max) = sync_status.peers();
-				let (status, message) = match connected {
-					0 => {
-						(HealthStatus::Bad, "You are not connected to any peers. There is most likely some network issue. Fix connectivity.".into())
-					},
-					1 => (HealthStatus::NeedsAttention, "You are connected to only one peer. Your node might not be reliable. Check your network connection.".into()),
-					_ => (HealthStatus::Ok, "".into()),
-				};
-				HealthInfo { status, message, details: (connected, max) }
-			};
-
-			// Check sync
-			let sync = {
-				let is_syncing = sync_status.is_major_importing();
-				let (status, message) = if is_syncing {
-					(HealthStatus::NeedsAttention, "Your node is still syncing, the values you see might be outdated. Wait until it's fully synced.".into())
-				} else {
-					(HealthStatus::Ok, "".into())
-				};
-				HealthInfo { status, message, details: is_syncing }
-			};
-
-			// Check time
-			let time = {
-				let (status, message, details) = match time {
-					Ok(Ok(diff)) if diff < MAX_DRIFT && diff > -MAX_DRIFT => {
-						(HealthStatus::Ok, "".into(), diff)
-					},
-					Ok(Ok(diff)) => {
-						(HealthStatus::Bad, format!(
-							"Your clock is not in sync. Detected difference is too big for the protocol to work: {}ms. Synchronize your clock.",
-							diff,
-						), diff)
-					},
-					Ok(Err(err)) => {
-						(HealthStatus::NeedsAttention, format!(
-							"Unable to reach time API: {}. Make sure that your clock is synchronized.",
-							err,
-						), 0)
-					},
-					Err(_) => {
-						(HealthStatus::NeedsAttention, "Time API request timed out. Make sure that the clock is synchronized.".into(), 0)
-					},
-				};
-
-				HealthInfo { status, message, details, }
-			};
-
-			Ok(Health { peers, sync, time})
-		}))
-	}
-}

dapps/node-health/src/lib.rs

@@ -1,49 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Node Health status reporting.
-
-#![warn(missing_docs)]
-
-extern crate futures;
-extern crate futures_cpupool;
-extern crate ntp;
-extern crate time as time_crate;
-extern crate parity_reactor;
-extern crate parking_lot;
-
-#[macro_use]
-extern crate log;
-#[macro_use]
-extern crate serde_derive;
-
-mod health;
-mod time;
-mod types;
-
-pub use futures_cpupool::CpuPool;
-pub use health::NodeHealth;
-pub use types::{Health, HealthInfo, HealthStatus};
-pub use time::{TimeChecker, Error};
-
-/// Indicates sync status
-pub trait SyncStatus: ::std::fmt::Debug + Send + Sync {
-	/// Returns true if there is a major sync happening.
-	fn is_major_importing(&self) -> bool;
-
-	/// Returns number of connected and ideal peers.
-	fn peers(&self) -> (usize, usize);
-}

dapps/node-health/src/time.rs

@@ -1,357 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Periodically checks node's time drift using [SNTP](https://tools.ietf.org/html/rfc1769).
-//!
-//! An NTP packet is sent to the server with a local timestamp, the server then completes the packet, yielding the
-//! following timestamps:
-//!
-//!    Timestamp Name          ID   When Generated
-//!   ------------------------------------------------------------
-//!    Originate Timestamp     T1   time request sent by client
-//!    Receive Timestamp       T2   time request received at server
-//!    Transmit Timestamp      T3   time reply sent by server
-//!    Destination Timestamp   T4   time reply received at client
-//!
-//! The drift is defined as:
-//!
-//! drift = ((T2 - T1) + (T3 - T4)) / 2.
-//!
-
-use std::io;
-use std::{fmt, mem, time};
-use std::collections::VecDeque;
-use std::sync::atomic::{self, AtomicUsize};
-use std::sync::Arc;
-
-use futures::{self, Future};
-use futures::future::{self, IntoFuture};
-use futures_cpupool::{CpuPool, CpuFuture};
-use ntp;
-use parking_lot::RwLock;
-use time_crate::{Duration, Timespec};
-
-/// Time checker error.
-#[derive(Debug, Clone, PartialEq)]
-pub enum Error {
-	/// No servers are currently available for a query.
-	NoServersAvailable,
-	/// There was an error when trying to reach the NTP server.
-	Ntp(String),
-	/// IO error when reading NTP response.
-	Io(String),
-}
-
-impl fmt::Display for Error {
-	fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
-		use self::Error::*;
-
-		match *self {
-			NoServersAvailable => write!(fmt, "No NTP servers available"),
-			Ntp(ref err) => write!(fmt, "NTP error: {}", err),
-			Io(ref err) => write!(fmt, "Connection Error: {}", err),
-		}
-	}
-}
-
-impl From<io::Error> for Error {
-	fn from(err: io::Error) -> Self { Error::Io(format!("{}", err)) }
-}
-
-impl From<ntp::errors::Error> for Error {
-	fn from(err: ntp::errors::Error) -> Self { Error::Ntp(format!("{}", err)) }
-}
-
-/// NTP time drift checker.
-pub trait Ntp {
-	/// Returned Future.
-	type Future: IntoFuture<Item=Duration, Error=Error>;
-
-	/// Returns the current time drift.
-	fn drift(&self) -> Self::Future;
-}
-
-const SERVER_MAX_POLL_INTERVAL_SECS: u64 = 60;
-#[derive(Debug)]
-struct Server {
-	pub address: String,
-	next_call: RwLock<time::Instant>,
-	failures: AtomicUsize,
-}
-
-impl Server {
-	pub fn is_available(&self) -> bool {
-		*self.next_call.read() < time::Instant::now()
-	}
-
-	pub fn report_success(&self) {
-		self.failures.store(0, atomic::Ordering::SeqCst);
-		self.update_next_call(1)
-	}
-
-	pub fn report_failure(&self) {
-		let errors = self.failures.fetch_add(1, atomic::Ordering::SeqCst);
-		self.update_next_call(1 << errors)
-	}
-
-	fn update_next_call(&self, delay: usize) {
-		*self.next_call.write() = time::Instant::now() + time::Duration::from_secs(delay as u64 * SERVER_MAX_POLL_INTERVAL_SECS);
-	}
-}
-
-impl<T: AsRef<str>> From<T> for Server {
-	fn from(t: T) -> Self {
-		Server {
-			address: t.as_ref().to_owned(),
-			next_call: RwLock::new(time::Instant::now()),
-			failures: Default::default(),
-		}
-	}
-}
-
-/// NTP client using the SNTP algorithm for calculating drift.
-#[derive(Clone)]
-pub struct SimpleNtp {
-	addresses: Vec<Arc<Server>>,
-	pool: CpuPool,
-}
-
-impl fmt::Debug for SimpleNtp {
-	fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
-		f
-			.debug_struct("SimpleNtp")
-			.field("addresses", &self.addresses)
-			.finish()
-	}
-}
-
-impl SimpleNtp {
-	fn new<T: AsRef<str>>(addresses: &[T], pool: CpuPool) -> SimpleNtp {
-		SimpleNtp {
-			addresses: addresses.iter().map(Server::from).map(Arc::new).collect(),
-			pool: pool,
-		}
-	}
-}
-
-impl Ntp for SimpleNtp {
-	type Future = future::Either<
-		CpuFuture<Duration, Error>,
-		future::FutureResult<Duration, Error>,
-	>;
-
-	fn drift(&self) -> Self::Future {
-		use self::future::Either::{A, B};
-
-		let server = self.addresses.iter().find(|server| server.is_available());
-		server.map(|server| {
-			let server = server.clone();
-			A(self.pool.spawn_fn(move || {
-				debug!(target: "dapps", "Fetching time from {}.", server.address);
-
-				match ntp::request(&server.address) {
-					Ok(packet) => {
-						let dest_time = ::time_crate::now_utc().to_timespec();
-						let orig_time = Timespec::from(packet.orig_time);
-						let recv_time = Timespec::from(packet.recv_time);
-						let transmit_time = Timespec::from(packet.transmit_time);
-
-						let drift = ((recv_time - orig_time) + (transmit_time - dest_time)) / 2;
-
-						server.report_success();
-						Ok(drift)
-					},
-					Err(err) => {
-						server.report_failure();
-						Err(err.into())
-					},
-				}
-			}))
-		}).unwrap_or_else(|| B(future::err(Error::NoServersAvailable)))
-	}
-}
-
-// NOTE In a positive scenario first results will be seen after:
-// MAX_RESULTS * UPDATE_TIMEOUT_INCOMPLETE_SECS seconds.
-const MAX_RESULTS: usize = 4;
-const UPDATE_TIMEOUT_OK_SECS: u64 = 6 * 60 * 60;
-const UPDATE_TIMEOUT_WARN_SECS: u64 = 15 * 60;
-const UPDATE_TIMEOUT_ERR_SECS: u64 = 60;
-const UPDATE_TIMEOUT_INCOMPLETE_SECS: u64 = 10;
-
-/// Maximal valid time drift.
-pub const MAX_DRIFT: i64 = 10_000;
-
-type BoxFuture<A, B> = Box<Future<Item = A, Error = B> + Send>;
-
-#[derive(Debug, Clone)]
-/// A time checker.
-pub struct TimeChecker<N: Ntp = SimpleNtp> {
-	ntp: N,
-	last_result: Arc<RwLock<(time::Instant, VecDeque<Result<i64, Error>>)>>,
-}
-
-impl TimeChecker<SimpleNtp> {
-	/// Creates new time checker given the NTP server address.
-	pub fn new<T: AsRef<str>>(ntp_addresses: &[T], pool: CpuPool) -> Self {
-		let last_result = Arc::new(RwLock::new(
-			// Assume everything is ok at the very beginning.
-			(time::Instant::now(), vec![Ok(0)].into())
-		));
-
-		let ntp = SimpleNtp::new(ntp_addresses, pool);
-
-		TimeChecker {
-			ntp,
-			last_result,
-		}
-	}
-}
-
-impl<N: Ntp> TimeChecker<N> where <N::Future as IntoFuture>::Future: Send + 'static {
-	/// Updates the time
-	pub fn update(&self) -> BoxFuture<i64, Error> {
-		trace!(target: "dapps", "Updating time from NTP.");
-		let last_result = self.last_result.clone();
-		Box::new(self.ntp.drift().into_future().then(move |res| {
-			let res = res.map(|d| d.num_milliseconds());
-
-			if let Err(Error::NoServersAvailable) = res {
-				debug!(target: "dapps", "No NTP servers available. Selecting an older result.");
-				return select_result(last_result.read().1.iter());
-			}
-
-			// Update the results.
-			let mut results = mem::replace(&mut last_result.write().1, VecDeque::new());
-			let has_all_results = results.len() >= MAX_RESULTS;
-			let valid_till = time::Instant::now() + time::Duration::from_secs(
-				match res {
-					Ok(time) if has_all_results && time < MAX_DRIFT => UPDATE_TIMEOUT_OK_SECS,
-					Ok(_) if has_all_results => UPDATE_TIMEOUT_WARN_SECS,
-					Err(_) if has_all_results => UPDATE_TIMEOUT_ERR_SECS,
-					_ => UPDATE_TIMEOUT_INCOMPLETE_SECS,
-				}
-			);
-
-			trace!(target: "dapps", "New time drift received: {:?}", res);
-			// Push the result.
-			results.push_back(res);
-			while results.len() > MAX_RESULTS {
-				results.pop_front();
-			}
-
-			// Select a response and update last result.
-			let res = select_result(results.iter());
-			*last_result.write() = (valid_till, results);
-			res
-		}))
-	}
-
-	/// Returns a current time drift or error if last request to NTP server failed.
-	pub fn time_drift(&self) -> BoxFuture<i64, Error> {
-		// return cached result
-		{
-			let res = self.last_result.read();
-			if res.0 > time::Instant::now() {
-				return Box::new(futures::done(select_result(res.1.iter())));
-			}
-		}
-		// or update and return result
-		self.update()
-	}
-}
-
-fn select_result<'a, T: Iterator<Item=&'a Result<i64, Error>>>(results: T) -> Result<i64, Error> {
-	let mut min = None;
-	for res in results {
-		min = Some(match (min.take(), res) {
-			(Some(Ok(min)), &Ok(ref new)) => Ok(::std::cmp::min(min, *new)),
-			(Some(Ok(old)), &Err(_)) => Ok(old),
-			(_, ref new) => (*new).clone(),
-		})
-	}
-
-	min.unwrap_or_else(|| Err(Error::Ntp("NTP server unavailable.".into())))
-}
-
-#[cfg(test)]
-mod tests {
-	use std::sync::Arc;
-	use std::cell::{Cell, RefCell};
-	use std::time::Instant;
-	use time::Duration;
-	use futures::{future, Future};
-	use super::{Ntp, TimeChecker, Error};
-	use parking_lot::RwLock;
-
-	#[derive(Clone)]
-	struct FakeNtp(RefCell<Vec<Duration>>, Cell<u64>);
-	impl FakeNtp {
-		fn new() -> FakeNtp {
-			FakeNtp(
-				RefCell::new(vec![Duration::milliseconds(150)]),
-				Cell::new(0))
-		}
-	}
-
-	impl Ntp for FakeNtp {
-		type Future = future::FutureResult<Duration, Error>;
-
-		fn drift(&self) -> Self::Future {
-			self.1.set(self.1.get() + 1);
-			future::ok(self.0.borrow_mut().pop().expect("Unexpected call to drift()."))
-		}
-	}
-
-	fn time_checker() -> TimeChecker<FakeNtp> {
-		let last_result = Arc::new(RwLock::new(
-			(Instant::now(), vec![Err(Error::Ntp("NTP server unavailable".into()))].into())
-		));
-
-		TimeChecker {
-			ntp: FakeNtp::new(),
-			last_result: last_result,
-		}
-	}
-
-	#[test]
-	fn should_fetch_time_on_start() {
-		// given
-		let time = time_checker();
-
-		// when
-		let diff = time.time_drift().wait().unwrap();
-
-		// then
-		assert_eq!(diff, 150);
-		assert_eq!(time.ntp.1.get(), 1);
-	}
-
-	#[test]
-	fn should_not_fetch_twice_if_timeout_has_not_passed() {
-		// given
-		let time = time_checker();
-
-		// when
-		let diff1 = time.time_drift().wait().unwrap();
-		let diff2 = time.time_drift().wait().unwrap();
-
-		// then
-		assert_eq!(diff1, 150);
-		assert_eq!(diff2, 150);
-		assert_eq!(time.ntp.1.get(), 1);
-	}
-}

dapps/node-health/src/types.rs

@@ -1,57 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Base health types.
-
-/// Health API endpoint status.
-#[derive(Debug, PartialEq, Serialize)]
-pub enum HealthStatus {
-	/// Everything's OK.
-	#[serde(rename = "ok")]
-	Ok,
-	/// Node health need attention
-	/// (the issue is not critical, but may need investigation)
-	#[serde(rename = "needsAttention")]
-	NeedsAttention,
-	/// There is something bad detected with the node.
-	#[serde(rename = "bad")]
-	Bad,
-}
-
-/// Represents a single check in node health.
-/// Cointains the status of that check and apropriate message and details.
-#[derive(Debug, PartialEq, Serialize)]
-#[serde(deny_unknown_fields)]
-pub struct HealthInfo<T> {
-	/// Check status.
-	pub status: HealthStatus,
-	/// Human-readable message.
-	pub message: String,
-	/// Technical details of the check.
-	pub details: T,
-}
-
-/// Node Health status.
-#[derive(Debug, PartialEq, Serialize)]
-#[serde(deny_unknown_fields)]
-pub struct Health {
-	/// Status of peers.
-	pub peers: HealthInfo<(usize, usize)>,
-	/// Sync status.
-	pub sync: HealthInfo<bool>,
-	/// Time diff info.
-	pub time: HealthInfo<i64>,
-}

dapps/src/api/api.rs

@@ -1,97 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use std::sync::Arc;
-
-use hyper::{Method, StatusCode};
-
-use api::response;
-use apps::fetcher::Fetcher;
-use endpoint::{Endpoint, Request, Response, EndpointPath};
-use futures::{future, Future};
-use node_health::{NodeHealth, HealthStatus};
-
-#[derive(Clone)]
-pub struct RestApi {
-	fetcher: Arc<Fetcher>,
-	health: NodeHealth,
-}
-
-impl Endpoint for RestApi {
-	fn respond(&self, mut path: EndpointPath, req: Request) -> Response {
-		if let Method::Options = *req.method() {
-			return Box::new(future::ok(response::empty()));
-		}
-
-		let endpoint = path.app_params.get(0).map(String::to_owned);
-		let hash = path.app_params.get(1).map(String::to_owned);
-
-		// at this point path.app_id contains 'api', adjust it to the hash properly, otherwise
-		// we will try and retrieve 'api' as the hash when doing the /api/content route
-		if let Some(ref hash) = hash {
-			path.app_id = hash.to_owned();
-		}
-
-		trace!(target: "dapps", "Handling /api request: {:?}/{:?}", endpoint, hash);
-		match endpoint.as_ref().map(String::as_str) {
-			Some("ping") => Box::new(future::ok(response::ping(req))),
-			Some("health") => self.health(),
-			Some("content") => self.resolve_content(hash.as_ref().map(String::as_str), path, req),
-			_ => Box::new(future::ok(response::not_found())),
-		}
-	}
-}
-
-impl RestApi {
-	pub fn new(
-		fetcher: Arc<Fetcher>,
-		health: NodeHealth,
-	) -> Box<Endpoint> {
-		Box::new(RestApi {
-			fetcher,
-			health,
-		})
-	}
-
-	fn resolve_content(&self, hash: Option<&str>, path: EndpointPath, req: Request) -> Response {
-		trace!(target: "dapps", "Resolving content: {:?} from path: {:?}", hash, path);
-		match hash {
-			Some(hash) if self.fetcher.contains(hash) => {
-				self.fetcher.respond(path, req)
-			},
-			_ => Box::new(future::ok(response::not_found())),
-		}
-	}
-
-	fn health(&self) -> Response {
-		Box::new(self.health.health()
-			.then(|health| {
-				let status = match health {
-					Ok(ref health) => {
-						if [&health.peers.status, &health.sync.status].iter().any(|x| *x != &HealthStatus::Ok) {
-							StatusCode::PreconditionFailed // HTTP 412
-						} else {
-							StatusCode::Ok // HTTP 200
-						}
-					},
-					_ => StatusCode::ServiceUnavailable, // HTTP 503
-				};
-
-				Ok(response::as_json(status, &health).into())
-			})
-		)
-	}
-}

dapps/src/api/mod.rs

@@ -1,23 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! REST API
-
-mod api;
-mod response;
-mod types;
-
-pub use self::api::RestApi;

dapps/src/api/response.rs

@@ -1,43 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use serde::Serialize;
-use serde_json;
-use hyper::{self, mime, StatusCode};
-
-use handlers::{ContentHandler, EchoHandler};
-
-pub fn empty() -> hyper::Response {
-	ContentHandler::ok("".into(), mime::TEXT_PLAIN).into()
-}
-
-pub fn as_json<T: Serialize>(status: StatusCode, val: &T) -> hyper::Response {
-	let json = serde_json::to_string(val)
-		.expect("serialization to string is infallible; qed");
-	ContentHandler::new(status, json, mime::APPLICATION_JSON).into()
-}
-
-pub fn ping(req: hyper::Request) -> hyper::Response {
-	EchoHandler::new(req).into()
-}
-
-pub fn not_found() -> hyper::Response {
-	as_json(StatusCode::NotFound, &::api::types::ApiError {
-		code: "404".into(),
-		title: "Not Found".into(),
-		detail: "Resource you requested has not been found.".into(),
-	})
-}

dapps/src/api/types.rs

@@ -1,27 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-/// A structure representing any error in REST API.
-#[derive(Debug, PartialEq, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct ApiError {
-	/// Error code.
-	pub code: String,
-	/// Human-readable error summary.
-	pub title: String,
-	/// More technical error details.
-	pub detail: String,
-}

dapps/src/apps/app.rs

@@ -1,39 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-#[derive(Debug, PartialEq, Clone, Serialize, Deserialize)]
-#[serde(deny_unknown_fields)]
-pub struct App {
-	pub id: Option<String>,
-	pub name: String,
-	pub description: String,
-	pub version: String,
-	pub author: String,
-	#[serde(rename="iconUrl")]
-	pub icon_url: String,
-	#[serde(rename="localUrl")]
-	pub local_url: Option<String>,
-	#[serde(rename="allowJsEval")]
-	pub allow_js_eval: Option<bool>,
-}
-
-impl App {
-	pub fn with_id(&self, id: &str) -> Self {
-		let mut app = self.clone();
-		app.id = Some(id.into());
-		app
-	}
-}

dapps/src/apps/cache.rs

@@ -1,128 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Fetchable Dapps support.
-
-use std::fs;
-
-use linked_hash_map::LinkedHashMap;
-use page::local;
-use handlers::FetchControl;
-
-pub enum ContentStatus {
-	Fetching(FetchControl),
-	Ready(local::Dapp),
-}
-
-#[derive(Default)]
-pub struct ContentCache {
-	cache: LinkedHashMap<String, ContentStatus>,
-}
-
-impl ContentCache {
-	pub fn insert(&mut self, content_id: String, status: ContentStatus) -> Option<ContentStatus> {
-		self.cache.insert(content_id, status)
-	}
-
-	pub fn remove(&mut self, content_id: &str) -> Option<ContentStatus> {
-		self.cache.remove(content_id)
-	}
-
-	pub fn get(&mut self, content_id: &str) -> Option<&mut ContentStatus> {
-		self.cache.get_refresh(content_id)
-	}
-
-	pub fn clear_garbage(&mut self, expected_size: usize) -> Vec<(String, ContentStatus)> {
-		let len = self.cache.len();
-
-		if len <= expected_size {
-			return Vec::new();
-		}
-
-		let mut removed = Vec::with_capacity(len - expected_size);
-
-		while self.cache.len() > expected_size {
-			let entry = self.cache.pop_front().expect("expected_size bounded at 0, len is greater; qed");
-
-			match entry.1 {
-				ContentStatus::Fetching(ref fetch) => {
-					trace!(target: "dapps", "Aborting {} because of limit.", entry.0);
-					// Mark as aborted
-					fetch.abort()
-				},
-				ContentStatus::Ready(ref endpoint) => {
-					trace!(target: "dapps", "Removing {} because of limit.", entry.0);
-					// Remove path (dir or file)
-					let res = fs::remove_dir_all(&endpoint.path()).or_else(|_| fs::remove_file(&endpoint.path()));
-					if let Err(e) = res {
-						warn!(target: "dapps", "Unable to remove dapp/content from cache: {:?}", e);
-					}
-				}
-			}
-
-			removed.push(entry);
-		}
-		removed
-	}
-
-	#[cfg(test)]
-	pub fn len(&self) -> usize {
-		self.cache.len()
-	}
-}
-
-#[cfg(test)]
-mod tests {
-	use super::*;
-
-	fn only_keys(data: Vec<(String, ContentStatus)>) -> Vec<String> {
-		data.into_iter().map(|x| x.0).collect()
-	}
-
-	#[test]
-	fn should_remove_least_recently_used() {
-		// given
-		let mut cache = ContentCache::default();
-		cache.insert("a".into(), ContentStatus::Fetching(Default::default()));
-		cache.insert("b".into(), ContentStatus::Fetching(Default::default()));
-		cache.insert("c".into(), ContentStatus::Fetching(Default::default()));
-
-		// when
-		let res = cache.clear_garbage(2);
-
-		// then
-		assert_eq!(cache.len(), 2);
-		assert_eq!(only_keys(res), vec!["a"]);
-	}
-
-	#[test]
-	fn should_update_lru_if_accessed() {
-		// given
-		let mut cache = ContentCache::default();
-		cache.insert("a".into(), ContentStatus::Fetching(Default::default()));
-		cache.insert("b".into(), ContentStatus::Fetching(Default::default()));
-		cache.insert("c".into(), ContentStatus::Fetching(Default::default()));
-
-		// when
-		cache.get("a");
-		let res = cache.clear_garbage(2);
-
-		// then
-		assert_eq!(cache.len(), 2);
-		assert_eq!(only_keys(res), vec!["b"]);
-	}
-
-}

dapps/src/apps/fetcher/installers.rs

@@ -1,270 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use zip;
-use std::{fs, fmt};
-use std::io::{self, Read, Write};
-use std::path::PathBuf;
-use ethereum_types::H256;
-use fetch;
-use futures_cpupool::CpuPool;
-use hash::keccak_pipe;
-use mime_guess::Mime;
-
-use apps::manifest::{MANIFEST_FILENAME, deserialize_manifest, serialize_manifest, Manifest};
-use handlers::{ContentValidator, ValidatorResponse};
-use page::{local, PageCache};
-
-type OnDone = Box<Fn(Option<local::Dapp>) + Send>;
-
-fn write_response_and_check_hash(
-	id: &str,
-	mut content_path: PathBuf,
-	filename: &str,
-	response: fetch::Response
-) -> Result<(fs::File, PathBuf), ValidationError> {
-	// try to parse id
-	let id = id.parse().map_err(|_| ValidationError::InvalidContentId)?;
-
-	// check if content exists
-	if content_path.exists() {
-		warn!(target: "dapps", "Overwriting existing content at 0x{:?}", id);
-		fs::remove_dir_all(&content_path)?
-	}
-
-	// create directory
-	fs::create_dir_all(&content_path)?;
-
-	// append filename
-	content_path.push(filename);
-
-	// Now write the response
-	let mut file = io::BufWriter::new(fs::File::create(&content_path)?);
-	let mut reader = io::BufReader::new(fetch::BodyReader::new(response));
-	let hash = keccak_pipe(&mut reader, &mut file)?;
-	let mut file = file.into_inner()?;
-	file.flush()?;
-
-	// Validate hash
-	if id == hash {
-		// The writing above changed the file Read position, which we need later. So we just create a new file handle
-		// here.
-		Ok((fs::File::open(&content_path)?, content_path))
-	} else {
-		Err(ValidationError::HashMismatch {
-			expected: id,
-			got: hash,
-		})
-	}
-}
-
-pub struct Content {
-	id: String,
-	mime: Mime,
-	content_path: PathBuf,
-	on_done: OnDone,
-	pool: CpuPool,
-}
-
-impl Content {
-	pub fn new(id: String, mime: Mime, content_path: PathBuf, on_done: OnDone, pool: CpuPool) -> Self {
-		Content {
-			id,
-			mime,
-			content_path,
-			on_done,
-			pool,
-		}
-	}
-}
-
-impl ContentValidator for Content {
-	type Error = ValidationError;
-
-	fn validate_and_install(self, response: fetch::Response) -> Result<ValidatorResponse, ValidationError> {
-		let pool = self.pool;
-		let id = self.id.clone();
-		let mime = self.mime;
-		let validate = move |content_path: PathBuf| {
-			// Create dir
-			let (_, content_path) = write_response_and_check_hash(&id, content_path, &id, response)?;
-
-			Ok(local::Dapp::single_file(pool, content_path, mime, PageCache::Enabled))
-		};
-
-		// Prepare path for a file
-		let content_path = self.content_path.join(&self.id);
-		// Make sure to always call on_done (even in case of errors)!
-		let result = validate(content_path.clone());
-		// remove the file if there was an error
-		if result.is_err() {
-			// Ignore errors since the file might not exist
-			let _ = fs::remove_dir_all(&content_path);
-		}
-		(self.on_done)(result.as_ref().ok().cloned());
-		result.map(ValidatorResponse::Local)
-	}
-}
-
-pub struct Dapp {
-	id: String,
-	dapps_path: PathBuf,
-	on_done: OnDone,
-	pool: CpuPool,
-}
-
-impl Dapp {
-	pub fn new(id: String, dapps_path: PathBuf, on_done: OnDone, pool: CpuPool) -> Self {
-		Dapp {
-			id,
-			dapps_path,
-			on_done,
-			pool,
-		}
-	}
-
-	fn find_manifest(zip: &mut zip::ZipArchive<fs::File>) -> Result<(Manifest, PathBuf), ValidationError> {
-		for i in 0..zip.len() {
-			let mut file = zip.by_index(i)?;
-
-			if !file.name().ends_with(MANIFEST_FILENAME) {
-				continue;
-			}
-
-			// try to read manifest
-			let mut manifest = String::new();
-			let manifest = file
-				.read_to_string(&mut manifest).ok()
-				.and_then(|_| deserialize_manifest(manifest).ok());
-
-			if let Some(manifest) = manifest {
-				let mut manifest_location = PathBuf::from(file.name());
-				manifest_location.pop(); // get rid of filename
-				return Ok((manifest, manifest_location));
-			}
-		}
-
-		Err(ValidationError::ManifestNotFound)
-	}
-}
-
-impl ContentValidator for Dapp {
-	type Error = ValidationError;
-
-	fn validate_and_install(self, response: fetch::Response) -> Result<ValidatorResponse, ValidationError> {
-		let id = self.id.clone();
-		let pool = self.pool;
-		let validate = move |dapp_path: PathBuf| {
-			let (file, zip_path) = write_response_and_check_hash(&id, dapp_path.clone(), &format!("{}.zip", id), response)?;
-			trace!(target: "dapps", "Opening dapp bundle at {:?}", zip_path);
-			// Unpack archive
-			let mut zip = zip::ZipArchive::new(file)?;
-			// First find manifest file
-			let (mut manifest, manifest_dir) = Self::find_manifest(&mut zip)?;
-			// Overwrite id to match hash
-			manifest.id = Some(id);
-
-			// Unpack zip
-			for i in 0..zip.len() {
-				let mut file = zip.by_index(i)?;
-				let is_dir = file.name().chars().rev().next() == Some('/');
-
-				let file_path = PathBuf::from(file.name());
-				let location_in_manifest_base = file_path.strip_prefix(&manifest_dir);
-				// Create files that are inside manifest directory
-				if let Ok(location_in_manifest_base) = location_in_manifest_base {
-					let p = dapp_path.join(location_in_manifest_base);
-					// Check if it's a directory
-					if is_dir {
-						fs::create_dir_all(p)?;
-					} else {
-						let mut target = fs::File::create(p)?;
-						io::copy(&mut file, &mut target)?;
-					}
-				}
-			}
-
-			// Remove zip
-			fs::remove_file(&zip_path)?;
-
-			// Write manifest
-			let manifest_str = serialize_manifest(&manifest).map_err(ValidationError::ManifestSerialization)?;
-			let manifest_path = dapp_path.join(MANIFEST_FILENAME);
-			let mut manifest_file = fs::File::create(manifest_path)?;
-			manifest_file.write_all(manifest_str.as_bytes())?;
-			// Create endpoint
-			let endpoint = local::Dapp::new(pool, dapp_path, manifest.into(), PageCache::Enabled);
-			Ok(endpoint)
-		};
-
-		// Prepare directory for dapp
-		let target = self.dapps_path.join(&self.id);
-		// Validate the dapp
-		let result = validate(target.clone());
-		// remove the file if there was an error
-		if result.is_err() {
-			// Ignore errors since the file might not exist
-			let _ = fs::remove_dir_all(&target);
-		}
-		(self.on_done)(result.as_ref().ok().cloned());
-		result.map(ValidatorResponse::Local)
-	}
-}
-
-#[derive(Debug)]
-pub enum ValidationError {
-	Io(io::Error),
-	Zip(zip::result::ZipError),
-	InvalidContentId,
-	ManifestNotFound,
-	ManifestSerialization(String),
-	HashMismatch { expected: H256, got: H256, },
-}
-
-impl fmt::Display for ValidationError {
-	fn fmt(&self, f: &mut fmt::Formatter) -> Result<(), fmt::Error> {
-		match *self {
-			ValidationError::Io(ref io) => write!(f, "Unexpected IO error occured: {:?}", io),
-			ValidationError::Zip(ref zip) => write!(f, "Unable to read ZIP archive: {:?}", zip),
-			ValidationError::InvalidContentId => write!(f, "ID is invalid. It should be 256 bits keccak hash of content."),
-			ValidationError::ManifestNotFound => write!(f, "Downloaded Dapp bundle did not contain valid manifest.json file."),
-			ValidationError::ManifestSerialization(ref err) => {
-				write!(f, "There was an error during Dapp Manifest serialization: {:?}", err)
-			},
-			ValidationError::HashMismatch { ref expected, ref got } => {
-				write!(f, "Hash of downloaded content did not match. Expected:{:?}, Got:{:?}.", expected, got)
-			},
-		}
-	}
-}
-
-impl From<io::Error> for ValidationError {
-	fn from(err: io::Error) -> Self {
-		ValidationError::Io(err)
-	}
-}
-
-impl From<zip::result::ZipError> for ValidationError {
-	fn from(err: zip::result::ZipError) -> Self {
-		ValidationError::Zip(err)
-	}
-}
-
-impl From<io::IntoInnerError<io::BufWriter<fs::File>>> for ValidationError {
-	fn from(err: io::IntoInnerError<io::BufWriter<fs::File>>) -> Self {
-		ValidationError::Io(err.into())
-	}
-}

dapps/src/apps/fetcher/mod.rs

@@ -1,329 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Fetchable Dapps support.
-//! Manages downloaded (cached) Dapps and downloads them when necessary.
-//! Uses `URLHint` to resolve addresses into Dapps bundle file location.
-
-mod installers;
-
-use std::{fs, env};
-use std::path::PathBuf;
-use std::sync::Arc;
-use futures::{future, Future};
-use futures_cpupool::CpuPool;
-use fetch::{Client as FetchClient, Fetch};
-use hash_fetch::urlhint::{URLHintContract, URLHint, URLHintResult};
-
-use hyper::StatusCode;
-
-use ethereum_types::H256;
-use {SyncStatus, random_filename};
-use parking_lot::Mutex;
-use page::local;
-use handlers::{ContentHandler, ContentFetcherHandler};
-use endpoint::{self, Endpoint, EndpointPath};
-use apps::cache::{ContentCache, ContentStatus};
-
-/// Limit of cached dapps/content
-const MAX_CACHED_DAPPS: usize = 20;
-
-pub trait Fetcher: Endpoint + 'static {
-	fn contains(&self, content_id: &str) -> bool;
-}
-
-pub struct ContentFetcher<F: Fetch = FetchClient, R: URLHint + 'static = URLHintContract> {
-	cache_path: PathBuf,
-	resolver: R,
-	cache: Arc<Mutex<ContentCache>>,
-	sync: Arc<SyncStatus>,
-	fetch: F,
-	pool: CpuPool,
-	only_content: bool,
-}
-
-impl<R: URLHint + 'static, F: Fetch> Drop for ContentFetcher<F, R> {
-	fn drop(&mut self) {
-		// Clear cache path
-		let _ = fs::remove_dir_all(&self.cache_path);
-	}
-}
-
-impl<R: URLHint + 'static, F: Fetch> ContentFetcher<F, R> {
-	pub fn new(
-		resolver: R,
-		sync: Arc<SyncStatus>,
-		fetch: F,
-		pool: CpuPool,
-	) -> Self {
-		let mut cache_path = env::temp_dir();
-		cache_path.push(random_filename());
-
-		ContentFetcher {
-			cache_path,
-			resolver,
-			sync,
-			cache: Arc::new(Mutex::new(ContentCache::default())),
-			fetch,
-			pool,
-			only_content: true,
-		}
-	}
-
-	pub fn allow_dapps(mut self, dapps: bool) -> Self {
-		self.only_content = !dapps;
-		self
-	}
-
-	fn not_found() -> endpoint::Response {
-		Box::new(future::ok(ContentHandler::error(
-			StatusCode::NotFound,
-			"Resource Not Found",
-			"Requested resource was not found.",
-			None,
-		).into()))
-	}
-
-	fn still_syncing() -> endpoint::Response {
-		Box::new(future::ok(ContentHandler::error(
-			StatusCode::ServiceUnavailable,
-			"Sync In Progress",
-			"Your node is still syncing. We cannot resolve any content before it's fully synced.",
-			Some("<a href=\"javascript:window.location.reload()\">Refresh</a>"),
-		).into()))
-	}
-
-	fn dapps_disabled() -> endpoint::Response {
-		Box::new(future::ok(ContentHandler::error(
-			StatusCode::ServiceUnavailable,
-			"Network Dapps Not Available",
-			"This interface doesn't support network dapps for security reasons.",
-			None,
-		).into()))
-	}
-
-	#[cfg(test)]
-	fn set_status(&self, content_id: &str, status: ContentStatus) {
-		self.cache.lock().insert(content_id.to_owned(), status);
-	}
-
-	// resolve contract call synchronously.
-	// TODO: port to futures-based hyper and make it all async.
-	fn resolve(&self, content_id: H256) -> Option<URLHintResult> {
-		self.resolver.resolve(content_id)
-			.wait()
-			.unwrap_or_else(|e| { warn!("Error resolving content-id: {}", e); None })
-	}
-}
-
-impl<R: URLHint + 'static, F: Fetch> Fetcher for ContentFetcher<F, R> {
-	fn contains(&self, content_id: &str) -> bool {
-		{
-			let mut cache = self.cache.lock();
-			// Check if we already have the app
-			if cache.get(content_id).is_some() {
-				return true;
-			}
-		}
-		// fallback to resolver
-		if let Ok(content_id) = content_id.parse() {
-			// if there is content or we are syncing return true
-			self.sync.is_major_importing() || self.resolve(content_id).is_some()
-		} else {
-			false
-		}
-	}
-}
-
-impl<R: URLHint + 'static, F: Fetch> Endpoint for ContentFetcher<F, R> {
-	fn respond(&self, path: EndpointPath, req: endpoint::Request) -> endpoint::Response {
-		let mut cache = self.cache.lock();
-		let content_id = path.app_id.clone();
-
-		let (new_status, handler) = {
-			let status = cache.get(&content_id);
-			match status {
-				// Just serve the content
-				Some(&mut ContentStatus::Ready(ref endpoint)) => {
-					(None, endpoint.to_response(&path))
-				},
-				// Content is already being fetched
-				Some(&mut ContentStatus::Fetching(ref fetch_control)) if !fetch_control.is_deadline_reached() => {
-					trace!(target: "dapps", "Content fetching in progress. Waiting...");
-					(None, fetch_control.to_response(path))
-				},
-				// We need to start fetching the content
-				_ => {
-					trace!(target: "dapps", "Content unavailable. Fetching... {:?}", content_id);
-					let content_hex = content_id.parse().expect("to_handler is called only when `contains` returns true.");
-					let content = self.resolve(content_hex);
-
-					let cache = self.cache.clone();
-					let id = content_id.clone();
-					let on_done = move |result: Option<local::Dapp>| {
-						let mut cache = cache.lock();
-						match result {
-							Some(endpoint) => cache.insert(id.clone(), ContentStatus::Ready(endpoint)),
-							// In case of error
-							None => cache.remove(&id),
-						};
-					};
-
-					match content {
-						// Don't serve dapps if we are still syncing (but serve content)
-						Some(URLHintResult::Dapp(_)) if self.sync.is_major_importing() => {
-							(None, Self::still_syncing())
-						},
-						Some(URLHintResult::Dapp(_)) if self.only_content => {
-							(None, Self::dapps_disabled())
-						},
-						Some(content) => {
-							let handler = match content {
-								URLHintResult::Dapp(dapp) => {
-									ContentFetcherHandler::new(
-										req.method(),
-										&dapp.url(),
-										path,
-										installers::Dapp::new(
-											content_id.clone(),
-											self.cache_path.clone(),
-											Box::new(on_done),
-											self.pool.clone(),
-										),
-										self.fetch.clone(),
-										self.pool.clone(),
-									)
-								},
-							    URLHintResult::GithubDapp(content) => {
-									ContentFetcherHandler::new(
-										req.method(),
-										&content.url,
-										path,
-										installers::Dapp::new(
-											content_id.clone(),
-											self.cache_path.clone(),
-											Box::new(on_done),
-											self.pool.clone(),
-										),
-										self.fetch.clone(),
-										self.pool.clone(),
-									)
-							    },
-								URLHintResult::Content(content) => {
-									ContentFetcherHandler::new(
-										req.method(),
-										&content.url,
-										path,
-										installers::Content::new(
-											content_id.clone(),
-											content.mime,
-											self.cache_path.clone(),
-											Box::new(on_done),
-											self.pool.clone(),
-										),
-										self.fetch.clone(),
-										self.pool.clone(),
-									)
-								},
-							};
-
-							(Some(ContentStatus::Fetching(handler.fetch_control())), Box::new(handler) as endpoint::Response)
-						},
-						None if self.sync.is_major_importing() => {
-							(None, Self::still_syncing())
-						},
-						None => {
-							// This may happen when sync status changes in between
-							// `contains` and `to_handler`
-							(None, Self::not_found())
-						},
-					}
-				},
-			}
-		};
-
-		if let Some(status) = new_status {
-			cache.clear_garbage(MAX_CACHED_DAPPS);
-			cache.insert(content_id, status);
-		}
-
-		handler
-	}
-}
-
-#[cfg(test)]
-mod tests {
-	use std::env;
-	use std::sync::Arc;
-	use fetch::Client;
-	use futures::{future, Future};
-	use hash_fetch::urlhint::{URLHint, URLHintResult};
-	use ethereum_types::H256;
-
-	use apps::cache::ContentStatus;
-	use endpoint::EndpointInfo;
-	use page::local;
-	use super::{ContentFetcher, Fetcher};
-	use {SyncStatus};
-
-	#[derive(Clone)]
-	struct FakeResolver;
-	impl URLHint for FakeResolver {
-		fn resolve(&self, _id: H256) -> Box<Future<Item = Option<URLHintResult>, Error = String> + Send> {
-			Box::new(future::ok(None))
-		}
-	}
-
-	#[derive(Debug)]
-	struct FakeSync(bool);
-	impl SyncStatus for FakeSync {
-		fn is_major_importing(&self) -> bool { self.0 }
-		fn peers(&self) -> (usize, usize) { (0, 5) }
-	}
-
-	#[test]
-	fn should_true_if_contains_the_app() {
-		// given
-		let pool = ::futures_cpupool::CpuPool::new(1);
-		let path = env::temp_dir();
-		let fetcher = ContentFetcher::new(
-			FakeResolver,
-			Arc::new(FakeSync(false)),
-			Client::new().unwrap(),
-			pool.clone(),
-		).allow_dapps(true);
-
-		let handler = local::Dapp::new(pool, path, EndpointInfo {
-			id: None,
-			name: "fake".into(),
-			description: "".into(),
-			version: "".into(),
-			author: "".into(),
-			icon_url: "".into(),
-			local_url: Some("".into()),
-			allow_js_eval: None,
-		}, Default::default());
-
-		// when
-		fetcher.set_status("test", ContentStatus::Ready(handler));
-		fetcher.set_status("test2", ContentStatus::Fetching(Default::default()));
-
-		// then
-		assert_eq!(fetcher.contains("test"), true);
-		assert_eq!(fetcher.contains("test2"), true);
-		assert_eq!(fetcher.contains("test3"), false);
-	}
-}

dapps/src/apps/fs.rs

@@ -1,136 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use std::collections::BTreeMap;
-use std::io;
-use std::io::Read;
-use std::fs;
-use std::path::{Path, PathBuf};
-use futures_cpupool::CpuPool;
-
-use apps::manifest::{MANIFEST_FILENAME, deserialize_manifest};
-use endpoint::{Endpoint, EndpointInfo};
-use page::{local, PageCache};
-
-struct LocalDapp {
-	id: String,
-	path: PathBuf,
-	info: EndpointInfo,
-}
-
-/// Tries to find and read manifest file in given `path` to extract `EndpointInfo`
-/// If manifest is not found sensible default `EndpointInfo` is returned based on given `name`.
-fn read_manifest(name: &str, mut path: PathBuf) -> EndpointInfo {
-	path.push(MANIFEST_FILENAME);
-
-	fs::File::open(path.clone())
-		.map_err(|e| format!("{:?}", e))
-		.and_then(|mut f| {
-			// Reat file
-			let mut s = String::new();
-			f.read_to_string(&mut s).map_err(|e| format!("{:?}", e))?;
-			// Try to deserialize manifest
-			deserialize_manifest(s)
-		})
-		.unwrap_or_else(|e| {
-			warn!(target: "dapps", "Cannot read manifest file at: {:?}. Error: {:?}", path, e);
-
-			EndpointInfo {
-				id: None,
-				name: name.into(),
-				description: name.into(),
-				version: "0.0.0".into(),
-				author: "?".into(),
-				icon_url: "icon.png".into(),
-				local_url: None,
-				allow_js_eval: Some(false),
-			}
-		})
-}
-
-/// Returns Dapp Id and Local Dapp Endpoint for given filesystem path.
-/// Parses the path to extract last component (for name).
-/// `None` is returned when path is invalid or non-existent.
-pub fn local_endpoint<P: AsRef<Path>>(path: P, pool: CpuPool) -> Option<(String, Box<local::Dapp>)> {
-	let path = path.as_ref().to_owned();
-	path.canonicalize().ok().and_then(|path| {
-		let name = path.file_name().and_then(|name| name.to_str());
-		name.map(|name| {
-			let dapp = local_dapp(name.into(), path.clone());
-			(dapp.id, Box::new(local::Dapp::new(
-				pool.clone(), dapp.path, dapp.info, PageCache::Disabled)
-			))
-		})
-	})
-}
-
-fn local_dapp(name: String, path: PathBuf) -> LocalDapp {
-	// try to get manifest file
-	let info = read_manifest(&name, path.clone());
-	LocalDapp {
-		id: name,
-		path: path,
-		info: info,
-	}
-}
-
-/// Returns endpoints for Local Dapps found for given filesystem path.
-/// Scans the directory and collects `local::Dapp`.
-pub fn local_endpoints<P: AsRef<Path>>(dapps_path: P, pool: CpuPool) -> BTreeMap<String, Box<Endpoint>> {
-	let mut pages = BTreeMap::<String, Box<Endpoint>>::new();
-	for dapp in local_dapps(dapps_path.as_ref()) {
-		pages.insert(
-			dapp.id,
-			Box::new(local::Dapp::new(pool.clone(), dapp.path, dapp.info, PageCache::Disabled))
-		);
-	}
-	pages
-}
-
-fn local_dapps(dapps_path: &Path) -> Vec<LocalDapp> {
-	let files = fs::read_dir(dapps_path);
-	if let Err(e) = files {
-		warn!(target: "dapps", "Unable to load local dapps from: {}. Reason: {:?}", dapps_path.display(), e);
-		return vec![];
-	}
-
-	let files = files.expect("Check is done earlier");
-	files.map(|dir| {
-			let entry = dir?;
-			let file_type = entry.file_type()?;
-
-			// skip files
-			if file_type.is_file() {
-				return Err(io::Error::new(io::ErrorKind::NotFound, "Not a file"));
-			}
-
-			// take directory name and path
-			entry.file_name().into_string()
-				.map(|name| (name, entry.path()))
-				.map_err(|e| {
-					info!(target: "dapps", "Unable to load dapp: {:?}. Reason: {:?}", entry.path(), e);
-					io::Error::new(io::ErrorKind::NotFound, "Invalid name")
-				})
-		})
-		.filter_map(|m| {
-			if let Err(ref e) = m {
-				debug!(target: "dapps", "Ignoring local dapp: {:?}", e);
-			}
-			m.ok()
-		})
-		.map(|(name, path)| local_dapp(name, path))
-		.collect()
-}

dapps/src/apps/manifest.rs

@@ -1,34 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use serde_json;
-pub use apps::App as Manifest;
-
-pub const MANIFEST_FILENAME: &'static str = "manifest.json";
-
-pub fn deserialize_manifest(manifest: String) -> Result<Manifest, String> {
-	let mut manifest = serde_json::from_str::<Manifest>(&manifest).map_err(|e| format!("{:?}", e))?;
-	if manifest.id.is_none() {
-		return Err("App 'id' is missing.".into());
-	}
-	manifest.allow_js_eval = Some(manifest.allow_js_eval.unwrap_or(false));
-
-	Ok(manifest)
-}
-
-pub fn serialize_manifest(manifest: &Manifest) -> Result<String, String> {
-	serde_json::to_string_pretty(manifest).map_err(|e| format!("{:?}", e))
-}

dapps/src/apps/mod.rs

@@ -1,70 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use std::path::PathBuf;
-use std::sync::Arc;
-
-use endpoint::Endpoints;
-use futures_cpupool::CpuPool;
-use proxypac::ProxyPac;
-use web::Web;
-use fetch::Fetch;
-use WebProxyTokens;
-
-mod app;
-mod cache;
-pub mod fs;
-pub mod fetcher;
-pub mod manifest;
-
-pub use self::app::App;
-
-pub const HOME_PAGE: &'static str = "home";
-pub const RPC_PATH: &'static str = "rpc";
-pub const API_PATH: &'static str = "api";
-pub const WEB_PATH: &'static str = "web";
-pub const URL_REFERER: &'static str = "__referer=";
-
-pub fn all_endpoints<F: Fetch>(
-	dapps_path: PathBuf,
-	extra_dapps: Vec<PathBuf>,
-	dapps_domain: &str,
-	web_proxy_tokens: Arc<WebProxyTokens>,
-	fetch: F,
-	pool: CpuPool,
-) -> (Vec<String>, Endpoints) {
-	// fetch fs dapps at first to avoid overwriting builtins
-	let mut pages = fs::local_endpoints(dapps_path.clone(), pool.clone());
-	let local_endpoints: Vec<String> = pages.keys().cloned().collect();
-	for path in extra_dapps {
-		if let Some((id, endpoint)) = fs::local_endpoint(path.clone(), pool.clone()) {
-			pages.insert(id, endpoint);
-		} else {
-			warn!(target: "dapps", "Ignoring invalid dapp at {}", path.display());
-		}
-	}
-
-	pages.insert(
-		"proxy".into(),
-		ProxyPac::boxed(dapps_domain.to_owned())
-	);
-	pages.insert(
-		WEB_PATH.into(),
-		Web::boxed(web_proxy_tokens.clone(), fetch.clone(), pool.clone())
-	);
-
-	(local_endpoints, pages)
-}

dapps/src/endpoint.rs

@@ -1,49 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! URL Endpoint traits
-
-use std::collections::BTreeMap;
-
-use futures::Future;
-use hyper;
-
-#[derive(Debug, PartialEq, Default, Clone)]
-pub struct EndpointPath {
-	pub app_id: String,
-	pub app_params: Vec<String>,
-	pub query: Option<String>,
-	pub host: String,
-	pub port: u16,
-	pub using_dapps_domains: bool,
-}
-
-impl EndpointPath {
-	pub fn has_no_params(&self) -> bool {
-		self.app_params.is_empty() || self.app_params.iter().all(|x| x.is_empty())
-	}
-}
-
-pub type EndpointInfo = ::apps::App;
-pub type Endpoints = BTreeMap<String, Box<Endpoint>>;
-pub type Response = Box<Future<Item=hyper::Response, Error=hyper::Error> + Send>;
-pub type Request = hyper::Request;
-
-pub trait Endpoint : Send + Sync {
-	fn info(&self) -> Option<&EndpointInfo> { None }
-
-	fn respond(&self, path: EndpointPath, req: Request) -> Response;
-}

dapps/src/error_tpl.html

@@ -1,102 +0,0 @@
-<!DOCTYPE html>
-<html>
-	<head>
-		<meta charset="utf-8">
-		<meta name="viewport" content="width=device-width">
-		<title>{title}</title>
-		<style>
-			:root, :root body {{
-				margin: 0;
-				padding: 0;
-				border: 0;
-				font-size: 100%;
-				font: inherit;
-				vertical-align: baseline;
-				background: rgb(95, 95, 95);
-				color: rgba(255, 255, 255, 0.75);
-				font-size: 16px;
-				font-family: sans-serif;
-				font-weight: 300;
-			}}
-
-			:root a, :root a:visited {{
-				text-decoration: none;
-				cursor: pointer;
-				color: rgb(0, 151, 167); /* #f80 */
-			}}
-
-			:root a:hover {{
-				color: rgb(0, 174, 193);
-			}}
-
-			h1,h2,h3,h4,h5,h6 {{
-				font-weight: 300;
-				text-transform: uppercase;
-				text-decoration: none;
-			}}
-
-			h1 {{
-				font-size: 24px;
-				line-height: 36px;
-				color: rgb(0, 151, 167);
-			}}
-
-			h2 {{
-				font-size: 20px;
-				line-height: 34px;
-			}}
-
-			code,kbd,pre,samp {{
-				font-family: monospace;
-			}}
-
-			.parity-navbar {{
-				background: rgb(65, 65, 65);
-				height: 72px;
-				padding: 0 1rem;
-				display: flex;
-				justify-content: space-between;
-			}}
-
-			.parity-status {{
-				clear: both;
-				padding: 1rem;
-				margin: 1rem 0;
-				text-align: right;
-				opacity: 0.75;
-			}}
-
-			.parity-box {{
-				margin: 1rem;
-				padding: 1rem;
-				background-color: rgb(48, 48, 48);
-				box-sizing: border-box;
-				box-shadow: rgba(0, 0, 0, 0.117647) 0px 1px 6px, rgba(0, 0, 0, 0.117647) 0px 1px 4px;
-				border-radius: 2px;
-				z-index: 1;
-				color: #aaa;
-			}}
-
-			.parity-box h1,
-			.parity-box h2,
-			.parity-box h3,
-			.parity-box h4,
-			.parity-box h5,
-			.parity-box h6 {{
-				margin: 0;
-			}}
-		</style>
-	</head>
-	<body>
-		<div class="parity-navbar">
-		</div>
-		<div class="parity-box">
-			<h1>{title}</h1>
-			<h3>{message}</h3>
-			<p><code>{details}</code></p>
-		</div>
-		<div class="parity-status">
-			<small>{version}</small>
-		</div>
-	</body>
-</html>

dapps/src/handlers/content.rs

@@ -1,79 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Simple Content Handler
-
-use hyper::{self, mime, header};
-use hyper::StatusCode;
-
-use parity_version::version;
-
-use handlers::add_security_headers;
-
-#[derive(Debug, Clone)]
-pub struct ContentHandler {
-	code: StatusCode,
-	content: String,
-	mimetype: mime::Mime,
-}
-
-impl ContentHandler {
-	pub fn ok(content: String, mimetype: mime::Mime) -> Self {
-		Self::new(StatusCode::Ok, content, mimetype)
-	}
-
-	pub fn html(code: StatusCode, content: String) -> Self {
-		Self::new(code, content, mime::TEXT_HTML)
-	}
-
-	pub fn error(
-		code: StatusCode,
-		title: &str,
-		message: &str,
-		details: Option<&str>,
-	) -> Self {
-		Self::html(code, format!(
-			include_str!("../error_tpl.html"),
-			title=title,
-			message=message,
-			details=details.unwrap_or_else(|| ""),
-			version=version(),
-		))
-	}
-
-	pub fn new(
-		code: StatusCode,
-		content: String,
-		mimetype: mime::Mime,
-	) -> Self {
-		ContentHandler {
-			code,
-			content,
-			mimetype,
-		}
-	}
-}
-
-impl Into<hyper::Response> for ContentHandler {
-	fn into(self) -> hyper::Response {
-		let mut res = hyper::Response::new()
-			.with_status(self.code)
-			.with_header(header::ContentType(self.mimetype))
-			.with_body(self.content);
-		add_security_headers(&mut res.headers_mut(), false);
-		res
-	}
-}

dapps/src/handlers/echo.rs

@@ -1,46 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Echo Handler
-
-use hyper::{self, header};
-
-use handlers::add_security_headers;
-
-#[derive(Debug)]
-pub struct EchoHandler {
-	request: hyper::Request,
-}
-
-impl EchoHandler {
-	pub fn new(request: hyper::Request) -> Self {
-		EchoHandler {
-			request,
-		}
-	}
-}
-
-impl Into<hyper::Response> for EchoHandler {
-	fn into(self) -> hyper::Response {
-		let content_type = self.request.headers().get().cloned();
-		let mut res = hyper::Response::new()
-			.with_header(content_type.unwrap_or(header::ContentType::json()))
-			.with_body(self.request.body());
-
-		add_security_headers(res.headers_mut(), false);
-		res
-	}
-}

dapps/src/handlers/errors.rs

@@ -1,66 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Handler errors.
-
-use handlers::{ContentHandler, FETCH_TIMEOUT};
-use hyper::StatusCode;
-use std::fmt;
-
-pub fn streaming() -> ContentHandler {
-	ContentHandler::error(
-		StatusCode::BadGateway,
-		"Streaming Error",
-		"This content is being streamed in other place.",
-		None,
-	)
-}
-
-pub fn download_error<E: fmt::Debug>(e: E) -> ContentHandler {
-	ContentHandler::error(
-		StatusCode::BadGateway,
-		"Download Error",
-		"There was an error when fetching the content.",
-		Some(&format!("{:?}", e)),
-	)
-}
-
-pub fn invalid_content<E: fmt::Debug>(e: E) -> ContentHandler {
-	ContentHandler::error(
-		StatusCode::BadGateway,
-		"Invalid Dapp",
-		"Downloaded bundle does not contain a valid content.",
-		Some(&format!("{:?}", e)),
-	)
-}
-
-pub fn timeout_error() -> ContentHandler {
-	ContentHandler::error(
-		StatusCode::GatewayTimeout,
-		"Download Timeout",
-		&format!("Could not fetch content within {} seconds.", FETCH_TIMEOUT.as_secs()),
-		None,
-	)
-}
-
-pub fn method_not_allowed() -> ContentHandler {
-	ContentHandler::error(
-		StatusCode::MethodNotAllowed,
-		"Method Not Allowed",
-		"Only <code>GET</code> requests are allowed.",
-		None,
-	)
-}

dapps/src/handlers/fetch.rs

@@ -1,305 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Hyper Server Handler that fetches a file during a request (proxy).
-
-use std::{fmt, mem};
-use std::sync::Arc;
-use std::sync::atomic::{AtomicBool, Ordering};
-use std::time::Instant;
-use fetch::{self, Fetch};
-use futures::sync::oneshot;
-use futures::{self, Future};
-use futures_cpupool::CpuPool;
-use hyper;
-use parking_lot::Mutex;
-
-use endpoint::{self, EndpointPath};
-use handlers::{ContentHandler, StreamingHandler, FETCH_TIMEOUT, errors};
-use page::local;
-
-pub enum ValidatorResponse {
-	Local(local::Dapp),
-	Streaming(StreamingHandler<fetch::BodyReader>),
-}
-
-pub trait ContentValidator: Sized + Send + 'static {
-	type Error: fmt::Debug + fmt::Display;
-
-	fn validate_and_install(self, fetch::Response) -> Result<ValidatorResponse, Self::Error>;
-}
-
-#[derive(Debug, Clone)]
-pub struct FetchControl {
-	abort: Arc<AtomicBool>,
-	listeners: Arc<Mutex<Vec<oneshot::Sender<WaitResult>>>>,
-	deadline: Instant,
-}
-
-impl Default for FetchControl {
-	fn default() -> Self {
-		FetchControl {
-			abort: Arc::new(AtomicBool::new(false)),
-			listeners: Arc::new(Mutex::new(Vec::new())),
-			deadline: Instant::now() + FETCH_TIMEOUT,
-		}
-	}
-}
-
-impl FetchControl {
-	pub fn is_deadline_reached(&self) -> bool {
-		self.deadline < Instant::now()
-	}
-
-	pub fn abort(&self) {
-		self.abort.store(true, Ordering::SeqCst);
-	}
-
-	pub fn to_response(&self, path: EndpointPath) -> endpoint::Response {
-		let (tx, receiver) = oneshot::channel();
-		self.listeners.lock().push(tx);
-
-		Box::new(WaitingHandler {
-			path,
-			state: WaitState::Waiting(receiver),
-		})
-	}
-
-	fn notify<F: Fn() -> WaitResult>(&self, status: F) {
-		let mut listeners = self.listeners.lock();
-		for sender in listeners.drain(..) {
-			trace!(target: "dapps", "Resuming request waiting for content...");
-			if let Err(_) = sender.send(status()) {
-				trace!(target: "dapps", "Waiting listener notification failed.");
-			}
-		}
-	}
-
-	fn set_status(&self, status: &FetchState) {
-		match *status {
-			FetchState::Error(ref handler) => self.notify(|| WaitResult::Error(handler.clone())),
-			FetchState::Done(ref endpoint, _) => self.notify(|| WaitResult::Done(endpoint.clone())),
-			FetchState::Streaming(_) => self.notify(|| WaitResult::NonAwaitable),
-			FetchState::InProgress(_) => {},
-			FetchState::Empty => {},
-		}
-	}
-}
-
-enum WaitState {
-	Waiting(oneshot::Receiver<WaitResult>),
-	Done(endpoint::Response),
-}
-
-#[derive(Debug)]
-enum WaitResult {
-	Error(ContentHandler),
-	Done(local::Dapp),
-	NonAwaitable,
-}
-
-pub struct WaitingHandler {
-	path: EndpointPath,
-	state: WaitState,
-}
-
-impl Future for WaitingHandler {
-	type Item = hyper::Response;
-	type Error = hyper::Error;
-
-	fn poll(&mut self) -> futures::Poll<Self::Item, Self::Error> {
-		loop {
-			let new_state = match self.state {
-				WaitState::Waiting(ref mut receiver) => {
-					let result = try_ready!(receiver.poll().map_err(|_| hyper::Error::Timeout));
-
-					match result {
-						WaitResult::Error(handler) => {
-							return Ok(futures::Async::Ready(handler.into()));
-						},
-						WaitResult::NonAwaitable => {
-							return Ok(futures::Async::Ready(errors::streaming().into()));
-						},
-						WaitResult::Done(endpoint) => {
-							WaitState::Done(endpoint.to_response(&self.path).into())
-						},
-					}
-				},
-				WaitState::Done(ref mut response) => {
-					return response.poll()
-				},
-			};
-
-			self.state = new_state;
-		}
-	}
-}
-
-enum FetchState {
-	Error(ContentHandler),
-	InProgress(Box<Future<Item=FetchState, Error=()> + Send>),
-	Streaming(hyper::Response),
-	Done(local::Dapp, endpoint::Response),
-	Empty,
-}
-
-impl fmt::Debug for FetchState {
-	fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
-		use self::FetchState::*;
-
-		write!(fmt, "FetchState(")?;
-		match *self {
-			Error(ref error) => write!(fmt, "error: {:?}", error),
-			InProgress(_) => write!(fmt, "in progress"),
-			Streaming(ref res) => write!(fmt, "streaming: {:?}", res),
-			Done(ref endpoint, _) => write!(fmt, "done: {:?}", endpoint),
-			Empty => write!(fmt, "?"),
-		}?;
-		write!(fmt, ")")
-	}
-}
-
-#[derive(Debug)]
-pub struct ContentFetcherHandler {
-	fetch_control: FetchControl,
-	status: FetchState,
-}
-
-impl ContentFetcherHandler {
-	pub fn fetch_control(&self) -> FetchControl {
-		self.fetch_control.clone()
-	}
-
-	pub fn new<H: ContentValidator, F: Fetch>(
-		method: &hyper::Method,
-		url: &str,
-		path: EndpointPath,
-		installer: H,
-		fetch: F,
-		pool: CpuPool,
-	) -> Self {
-		let fetch_control = FetchControl::default();
-
-		// Validation of method
-		let status = match *method {
-			// Start fetching content
-			hyper::Method::Get => {
-				trace!(target: "dapps", "Fetching content from: {:?}", url);
-				FetchState::InProgress(Self::fetch_content(
-						pool,
-						fetch,
-						url,
-						fetch_control.abort.clone(),
-						path,
-						installer,
-				))
-			},
-			// or return error
-			_ => FetchState::Error(errors::method_not_allowed()),
-		};
-
-		ContentFetcherHandler {
-			fetch_control,
-			status,
-		}
-	}
-
-	fn fetch_content<H: ContentValidator, F: Fetch>(
-		pool: CpuPool,
-		fetch: F,
-		url: &str,
-		abort: Arc<AtomicBool>,
-		path: EndpointPath,
-		installer: H,
-	) -> Box<Future<Item=FetchState, Error=()> + Send> {
-		// Start fetching the content
-		let pool2 = pool.clone();
-		let future = fetch.get(url, abort.into()).then(move |result| {
-			trace!(target: "dapps", "Fetching content finished. Starting validation: {:?}", result);
-			Ok(match result {
-				Ok(response) => match installer.validate_and_install(response) {
-					Ok(ValidatorResponse::Local(endpoint)) => {
-						trace!(target: "dapps", "Validation OK. Returning response.");
-						let response = endpoint.to_response(&path);
-						FetchState::Done(endpoint, response)
-					},
-					Ok(ValidatorResponse::Streaming(stream)) => {
-						trace!(target: "dapps", "Validation OK. Streaming response.");
-						let (reading, response) = stream.into_response();
-						pool.spawn(reading).forget();
-						FetchState::Streaming(response)
-					},
-					Err(e) => {
-						trace!(target: "dapps", "Error while validating content: {:?}", e);
-						FetchState::Error(errors::invalid_content(e))
-					},
-				},
-				Err(e) => {
-					warn!(target: "dapps", "Unable to fetch content: {:?}", e);
-					FetchState::Error(errors::download_error(e))
-				},
-			})
-		});
-
-		// make sure to run within fetch thread pool.
-		Box::new(pool2.spawn(future))
-	}
-}
-
-impl Future for ContentFetcherHandler {
-	type Item = hyper::Response;
-	type Error = hyper::Error;
-
-	fn poll(&mut self) -> futures::Poll<Self::Item, Self::Error> {
-		loop {
-			trace!(target: "dapps", "Polling status: {:?}", self.status);
-			self.status = match mem::replace(&mut self.status, FetchState::Empty) {
-				FetchState::Error(error) => {
-					return Ok(futures::Async::Ready(error.into()));
-				},
-				FetchState::Streaming(response) => {
-					return Ok(futures::Async::Ready(response));
-				},
-				any => any,
-			};
-
-			let status = match self.status {
-				// Request may time out
-				FetchState::InProgress(_) if self.fetch_control.is_deadline_reached() => {
-					trace!(target: "dapps", "Fetching dapp failed because of timeout.");
-					FetchState::Error(errors::timeout_error())
-				},
-				FetchState::InProgress(ref mut receiver) => {
-					// Check if there is a response
-					trace!(target: "dapps", "Polling streaming response.");
-					try_ready!(receiver.poll().map_err(|err| {
-						warn!(target: "dapps", "Error while fetching response: {:?}", err);
-						hyper::Error::Timeout
-					}))
-				},
-				FetchState::Done(_, ref mut response) => {
-					return response.poll()
-				},
-				FetchState::Empty => panic!("Future polled twice."),
-				_ => unreachable!(),
-			};
-
-			trace!(target: "dapps", "New status: {:?}", status);
-			self.fetch_control.set_status(&status);
-			self.status = status;
-		}
-	}
-}

dapps/src/handlers/mod.rs

@@ -1,91 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Hyper handlers implementations.
-
-mod content;
-mod echo;
-mod fetch;
-mod reader;
-mod redirect;
-mod streaming;
-mod errors;
-
-pub use self::content::ContentHandler;
-pub use self::echo::EchoHandler;
-pub use self::fetch::{ContentFetcherHandler, ContentValidator, FetchControl, ValidatorResponse};
-pub use self::reader::Reader;
-pub use self::redirect::Redirection;
-pub use self::streaming::StreamingHandler;
-
-use hyper::header;
-use std::time::Duration;
-
-const FETCH_TIMEOUT: Duration = Duration::from_secs(300);
-
-/// Adds security-related headers to the Response.
-pub fn add_security_headers(headers: &mut header::Headers, allow_js_eval: bool) {
-	headers.set_raw("X-XSS-Protection", "1; mode=block");
-	headers.set_raw("X-Content-Type-Options", "nosniff");
-	headers.set_raw("X-Frame-Options", "SAMEORIGIN");
-
-	// Content Security Policy headers
-	headers.set_raw("Content-Security-Policy", String::new()
-		// Restrict everything to the same origin by default.
-		+ "default-src 'self';"
-		// Allow connecting to WS servers and HTTP(S) servers.
-		// We could be more restrictive and allow only RPC server URL.
-		+ "connect-src http: https: ws: wss:;"
-		// Allow framing any content from HTTP(S).
-		// Again we could only allow embedding from RPC server URL.
-		// (deprecated)
-		+ "frame-src 'self' http: https:;"
-		// Allow framing and web workers from HTTP(S).
-		+ "child-src 'self' http: https:;"
-		// We allow data: blob: and HTTP(s) images.
-		// We could get rid of wildcarding HTTP and only allow RPC server URL.
-		// (http required for local dapps icons)
-		+ "img-src 'self' 'unsafe-inline' data: blob: http: https:;"
-		// Allow style from data: blob: and HTTPS.
-		+ "style-src 'self' 'unsafe-inline' data: blob: https:;"
-		// Allow fonts from data: and HTTPS.
-		+ "font-src 'self' data: https:;"
-		// Disallow objects
-		+ "object-src 'none';"
-		// Allow scripts
-		+ {
-			let script_src = "";
-			let eval = if allow_js_eval { " 'unsafe-eval'" } else { "" };
-
-			&format!(
-				"script-src 'self' {}{};",
-				script_src,
-				eval
-			)
-		}
-		// Same restrictions as script-src with additional
-		// blob: that is required for camera access (worker)
-		+ "worker-src 'self' https: blob:;"
-		// Run in sandbox mode (although it's not fully safe since we allow same-origin and script)
-		+ "sandbox allow-same-origin allow-forms allow-modals allow-popups allow-presentation allow-scripts;"
-		// Disallow submitting forms from any dapps
-		+ "form-action 'none';"
-		// Never allow mixed content
-		+ "block-all-mixed-content;"
-		// Specify if the site can be embedded.
-		+ "frame-ancestors 'self';"
-	);
-}

dapps/src/handlers/reader.rs

@@ -1,73 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! A chunk-producing io::Read wrapper.
-
-use std::io::{self, Read};
-
-use futures::{self, sink, Sink, Future};
-use futures::sync::mpsc;
-use hyper;
-
-type Sender = mpsc::Sender<Result<hyper::Chunk, hyper::Error>>;
-
-const MAX_CHUNK_SIZE: usize = 32 * 1024;
-
-/// A Reader is essentially a stream of `hyper::Chunks`.
-/// The chunks are read from given `io::Read` instance.
-///
-/// Unfortunately `hyper` doesn't allow you to pass `Stream`
-/// directly to the response, so you need to create
-/// a `Body::pair()` and send over chunks using `sink::Send`.
-/// Also `Chunks` need to take `Vec` by value, so we need
-/// to allocate it for each chunk being sent.
-pub struct Reader<R: io::Read> {
-	buffer: [u8; MAX_CHUNK_SIZE],
-	content: io::BufReader<R>,
-	sending: sink::Send<Sender>,
-}
-
-impl<R: io::Read> Reader<R> {
-	pub fn pair(content: R, initial: Vec<u8>) -> (Self, hyper::Body) {
-		let (tx, rx) = hyper::Body::pair();
-		let reader = Reader {
-			buffer: [0; MAX_CHUNK_SIZE],
-			content: io::BufReader::new(content),
-			sending: tx.send(Ok(initial.into())),
-		};
-
-		(reader, rx)
-	}
-}
-
-impl<R: io::Read> Future for Reader<R> {
-	type Item = ();
-	type Error = ();
-
-	fn poll(&mut self) -> futures::Poll<Self::Item, Self::Error> {
-		loop {
-			let next = try_ready!(self.sending.poll().map_err(|err| {
-				warn!(target: "dapps", "Unable to send next chunk: {:?}", err);
-			}));
-
-			self.sending = match self.content.read(&mut self.buffer) {
-				Ok(0) => return Ok(futures::Async::Ready(())),
-				Ok(read) => next.send(Ok(self.buffer[..read].to_vec().into())),
-				Err(err) => next.send(Err(hyper::Error::Io(err))),
-			}
-		}
-	}
-}

dapps/src/handlers/redirect.rs

@@ -1,41 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! HTTP Redirection hyper handler
-
-use hyper::{self, header, StatusCode};
-
-#[derive(Clone)]
-pub struct Redirection {
-	to_url: String
-}
-
-impl Redirection {
-	pub fn new<T: Into<String>>(url: T) -> Self {
-		Redirection {
-			to_url: url.into()
-		}
-	}
-}
-
-impl Into<hyper::Response> for Redirection {
-	fn into(self) -> hyper::Response {
-		// Don't use `MovedPermanently` here to prevent browser from caching the redirections.
-		hyper::Response::new()
-			.with_status(StatusCode::Found)
-			.with_header(header::Location::new(self.to_url))
-	}
-}

dapps/src/handlers/streaming.rs

@@ -1,55 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Content Stream Response
-
-use std::io;
-use hyper::{self, header, mime, StatusCode};
-
-use handlers::{add_security_headers, Reader};
-
-pub struct StreamingHandler<R> {
-	initial: Vec<u8>,
-	content: R,
-	status: StatusCode,
-	mimetype: mime::Mime,
-}
-
-impl<R: io::Read> StreamingHandler<R> {
-	pub fn new(content: R, status: StatusCode, mimetype: mime::Mime) -> Self {
-		StreamingHandler {
-			initial: Vec::new(),
-			content,
-			status,
-			mimetype,
-		}
-	}
-
-	pub fn set_initial_content(&mut self, content: &str) {
-		self.initial = content.as_bytes().to_vec();
-	}
-
-	pub fn into_response(self) -> (Reader<R>, hyper::Response) {
-		let (reader, body) = Reader::pair(self.content, self.initial);
-		let mut res = hyper::Response::new()
-			.with_status(self.status)
-			.with_header(header::ContentType(self.mimetype))
-			.with_body(body);
-		add_security_headers(&mut res.headers_mut(), false);
-
-		(reader, res)
-	}
-}

dapps/src/lib.rs

@@ -1,228 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Ethcore Webapplications for Parity
-#![warn(missing_docs)]
-
-extern crate base32;
-extern crate futures_cpupool;
-extern crate itertools;
-extern crate linked_hash_map;
-extern crate mime_guess;
-extern crate parking_lot;
-extern crate rand;
-extern crate rustc_hex;
-extern crate serde;
-extern crate serde_json;
-extern crate unicase;
-extern crate zip;
-
-extern crate jsonrpc_http_server;
-
-extern crate ethcore_bytes as bytes;
-extern crate ethereum_types;
-extern crate fetch;
-extern crate node_health;
-extern crate parity_dapps_glue as parity_dapps;
-extern crate parity_hash_fetch as hash_fetch;
-extern crate keccak_hash as hash;
-extern crate parity_version;
-extern crate registrar;
-
-#[macro_use]
-extern crate futures;
-#[macro_use]
-extern crate log;
-#[macro_use]
-extern crate serde_derive;
-
-#[cfg(test)]
-extern crate env_logger;
-#[cfg(test)]
-extern crate ethcore_devtools as devtools;
-#[cfg(test)]
-extern crate jsonrpc_core;
-#[cfg(test)]
-extern crate parity_reactor;
-
-mod endpoint;
-mod apps;
-mod page;
-mod router;
-mod handlers;
-mod api;
-mod proxypac;
-mod web;
-#[cfg(test)]
-mod tests;
-
-use std::collections::HashMap;
-use std::mem;
-use std::path::PathBuf;
-use std::sync::Arc;
-use futures_cpupool::CpuPool;
-use jsonrpc_http_server::{self as http, hyper, Origin};
-use parking_lot::RwLock;
-
-use fetch::Fetch;
-use node_health::NodeHealth;
-
-pub use registrar::{RegistrarClient, Asynchronous};
-pub use node_health::SyncStatus;
-pub use page::builtin::Dapp;
-
-/// Validates Web Proxy tokens
-pub trait WebProxyTokens: Send + Sync {
-	/// Should return a domain allowed to be accessed by this token or `None` if the token is not valid
-	fn domain(&self, token: &str) -> Option<Origin>;
-}
-
-impl<F> WebProxyTokens for F where F: Fn(String) -> Option<Origin> + Send + Sync {
-	fn domain(&self, token: &str) -> Option<Origin> { self(token.to_owned()) }
-}
-
-/// Current supported endpoints.
-#[derive(Default, Clone)]
-pub struct Endpoints {
-	local_endpoints: Arc<RwLock<Vec<String>>>,
-	endpoints: Arc<RwLock<endpoint::Endpoints>>,
-	dapps_path: PathBuf,
-	pool: Option<CpuPool>,
-}
-
-impl Endpoints {
-	/// Returns a current list of app endpoints.
-	pub fn list(&self) -> Vec<apps::App> {
-		self.endpoints.read().iter().filter_map(|(ref k, ref e)| {
-			e.info().map(|ref info| info.with_id(k))
-		}).collect()
-	}
-
-	/// Check for any changes in the local dapps folder and update.
-	pub fn refresh_local_dapps(&self) {
-		let pool = match self.pool.as_ref() {
-			None => return,
-			Some(pool) => pool,
-		};
-		let new_local = apps::fs::local_endpoints(&self.dapps_path, pool.clone());
-		let old_local = mem::replace(&mut *self.local_endpoints.write(), new_local.keys().cloned().collect());
-		let (_, to_remove): (_, Vec<_>) = old_local
-			.into_iter()
-			.partition(|k| new_local.contains_key(&k.clone()));
-
-		let mut endpoints = self.endpoints.write();
-		// remove the dead dapps
-		for k in to_remove {
-			endpoints.remove(&k);
-		}
-		// new dapps to be added
-		for (k, v) in new_local {
-			if !endpoints.contains_key(&k) {
-				endpoints.insert(k, v);
-			}
-		}
-	}
-}
-
-/// Dapps server as `jsonrpc-http-server` request middleware.
-pub struct Middleware {
-	endpoints: Endpoints,
-	router: router::Router,
-}
-
-impl Middleware {
-	/// Get local endpoints handle.
-	pub fn endpoints(&self) -> &Endpoints {
-		&self.endpoints
-	}
-
-	/// Creates new Dapps server middleware.
-	pub fn dapps<F: Fetch>(
-		pool: CpuPool,
-		health: NodeHealth,
-		dapps_path: PathBuf,
-		extra_dapps: Vec<PathBuf>,
-		dapps_domain: &str,
-		registrar: Arc<RegistrarClient<Call=Asynchronous>>,
-		sync_status: Arc<SyncStatus>,
-		web_proxy_tokens: Arc<WebProxyTokens>,
-		fetch: F,
-	) -> Self {
-		let content_fetcher = Arc::new(apps::fetcher::ContentFetcher::new(
-			hash_fetch::urlhint::URLHintContract::new(registrar),
-			sync_status.clone(),
-			fetch.clone(),
-			pool.clone(),
-		).allow_dapps(true));
-		let (local_endpoints, endpoints) = apps::all_endpoints(
-			dapps_path.clone(),
-			extra_dapps,
-			dapps_domain,
-			web_proxy_tokens,
-			fetch.clone(),
-			pool.clone(),
-		);
-		let endpoints = Endpoints {
-			endpoints: Arc::new(RwLock::new(endpoints)),
-			dapps_path,
-			local_endpoints: Arc::new(RwLock::new(local_endpoints)),
-			pool: Some(pool.clone()),
-		};
-
-		let special = special_endpoints(
-			health,
-			content_fetcher.clone(),
-		);
-
-		let router = router::Router::new(
-			content_fetcher,
-			Some(endpoints.clone()),
-			special,
-			dapps_domain.to_owned(),
-		);
-
-		Middleware {
-			endpoints,
-			router,
-		}
-	}
-}
-
-impl http::RequestMiddleware for Middleware {
-	fn on_request(&self, req: hyper::Request) -> http::RequestMiddlewareAction {
-		self.router.on_request(req)
-	}
-}
-
-fn special_endpoints(
-	health: NodeHealth,
-	content_fetcher: Arc<apps::fetcher::Fetcher>,
-) -> HashMap<router::SpecialEndpoint, Option<Box<endpoint::Endpoint>>> {
-	let mut special = HashMap::new();
-	special.insert(router::SpecialEndpoint::Rpc, None);
-	special.insert(router::SpecialEndpoint::Api, Some(api::RestApi::new(
-		content_fetcher,
-		health,
-	)));
-	special
-}
-
-/// Random filename
-fn random_filename() -> String {
-	use ::rand::Rng;
-	let mut rng = ::rand::OsRng::new().unwrap();
-	rng.gen_ascii_chars().take(12).collect()
-}

dapps/src/page/builtin.rs

@@ -1,145 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use std::io;
-use futures::future;
-use futures_cpupool::CpuPool;
-use hyper::mime::{self, Mime};
-use itertools::Itertools;
-use parity_dapps::{WebApp, Info};
-
-use endpoint::{Endpoint, EndpointInfo, EndpointPath, Request, Response};
-use page::{handler, PageCache};
-
-/// Represents a builtin Dapp.
-pub struct Dapp<T: WebApp + 'static> {
-	/// futures cpu pool
-	pool: CpuPool,
-	/// Content of the files
-	app: T,
-	info: EndpointInfo,
-	fallback_to_index_html: bool,
-}
-
-impl<T: WebApp + 'static> Dapp<T> {
-	/// Creates new `Dapp` for builtin (compile time) Dapp.
-	pub fn new(pool: CpuPool, app: T) -> Self {
-		let info = app.info();
-		Dapp {
-			pool,
-			app,
-			info: EndpointInfo::from(info),
-			fallback_to_index_html: false,
-		}
-	}
-
-	/// Creates a new `Dapp` for builtin (compile time) Dapp.
-	/// Instead of returning 404 this endpoint will always server index.html.
-	pub fn with_fallback_to_index(pool: CpuPool, app: T) -> Self {
-		let info = app.info();
-		Dapp {
-			pool,
-			app,
-			info: EndpointInfo::from(info),
-			fallback_to_index_html: true,
-		}
-	}
-
-	/// Allow the dapp to use `unsafe-eval` to run JS.
-	pub fn allow_js_eval(&mut self) {
-		self.info.allow_js_eval = Some(true);
-	}
-}
-
-impl<T: WebApp> Endpoint for Dapp<T> {
-	fn info(&self) -> Option<&EndpointInfo> {
-		Some(&self.info)
-	}
-
-	fn respond(&self, path: EndpointPath, _req: Request) -> Response {
-		trace!(target: "dapps", "Builtin file path: {:?}", path);
-		let file_path = if path.has_no_params() {
-			"index.html".to_owned()
-		} else {
-			path.app_params.into_iter().filter(|x| !x.is_empty()).join("/")
-		};
-		trace!(target: "dapps", "Builtin file: {:?}", file_path);
-
-		let file = {
-			let file = |path| self.app.file(path).map(|file| {
-				let content_type = match file.content_type.parse() {
-					Ok(mime) => mime,
-					Err(_) => {
-						warn!(target: "dapps", "invalid MIME type: {}", file.content_type);
-						mime::TEXT_HTML
-					},
-				};
-				BuiltinFile {
-					content_type,
-					content: io::Cursor::new(file.content),
-				}
-			});
-			let res = file(&file_path);
-			if self.fallback_to_index_html {
-				res.or_else(|| file("index.html"))
-			} else {
-				res
-			}
-		};
-
-		let (reader, response) = handler::PageHandler {
-			file,
-			cache: PageCache::Disabled,
-			allow_js_eval: self.info.allow_js_eval.clone().unwrap_or(false),
-		}.into_response();
-
-		self.pool.spawn(reader).forget();
-
-		Box::new(future::ok(response))
-	}
-}
-
-impl From<Info> for EndpointInfo {
-	fn from(info: Info) -> Self {
-		EndpointInfo {
-			id: None,
-			name: info.name.into(),
-			description: info.description.into(),
-			author: info.author.into(),
-			icon_url: info.icon_url.into(),
-			local_url: None,
-			version: info.version.into(),
-			allow_js_eval: None,
-		}
-	}
-}
-
-struct BuiltinFile {
-	content_type: Mime,
-	content: io::Cursor<&'static [u8]>,
-}
-
-impl handler::DappFile for BuiltinFile {
-	type Reader = io::Cursor<&'static [u8]>;
-
-	fn content_type(&self) -> &Mime {
-		&self.content_type
-	}
-
-	fn into_reader(self) -> Self::Reader {
-		self.content
-	}
-}

dapps/src/page/handler.rs

@@ -1,100 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use std::io;
-use std::time::{Duration, SystemTime};
-use hyper::{self, header, StatusCode};
-use hyper::mime::{Mime};
-
-use handlers::{Reader, ContentHandler, add_security_headers};
-
-/// Represents a file that can be sent to client.
-/// Implementation should keep track of bytes already sent internally.
-pub trait DappFile {
-	/// A reader type returned by this file.
-	type Reader: io::Read;
-
-	/// Returns a content-type of this file.
-	fn content_type(&self) -> &Mime;
-
-	/// Convert this file into io::Read instance.
-	fn into_reader(self) -> Self::Reader where Self: Sized;
-}
-
-/// Defines what cache headers should be appended to returned resources.
-#[derive(Debug, Copy, Clone, PartialEq)]
-pub enum PageCache {
-	Enabled,
-	Disabled,
-}
-
-impl Default for PageCache {
-	fn default() -> Self {
-		PageCache::Disabled
-	}
-}
-
-/// A handler for a single webapp.
-/// Resolves correct paths and serves as a plumbing code between
-/// hyper server and dapp.
-pub struct PageHandler<T: DappFile> {
-	/// File currently being served
-	pub file: Option<T>,
-	/// Cache settings for this page.
-	pub cache: PageCache,
-	/// Allow JS unsafe-eval.
-	pub allow_js_eval: bool,
-}
-
-impl<T: DappFile> PageHandler<T> {
-	pub fn into_response(self) -> (Option<Reader<T::Reader>>, hyper::Response) {
-		let file = match self.file {
-			None => return (None, ContentHandler::error(
-				StatusCode::NotFound,
-				"File not found",
-				"Requested file has not been found.",
-				None,
-			).into()),
-			Some(file) => file,
-		};
-
-		let mut res = hyper::Response::new()
-			.with_status(StatusCode::Ok);
-
-		// headers
-		{
-			let mut headers = res.headers_mut();
-
-			if let PageCache::Enabled = self.cache {
-				let validity_secs = 365u32 * 24 * 3600;
-				let validity = Duration::from_secs(validity_secs as u64);
-				headers.set(header::CacheControl(vec![
-					header::CacheDirective::Public,
-					header::CacheDirective::MaxAge(validity_secs),
-				]));
-				headers.set(header::Expires(header::HttpDate::from(SystemTime::now() + validity)));
-			}
-
-			headers.set(header::ContentType(file.content_type().to_owned()));
-
-			add_security_headers(&mut headers, self.allow_js_eval);
-		}
-
-		let (reader, body) = Reader::pair(file.into_reader(), Vec::new());
-		res.set_body(body);
-		(Some(reader), res)
-	}
-}

dapps/src/page/local.rs

@@ -1,141 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use mime_guess;
-use std::{fs, fmt};
-use std::path::{Path, PathBuf};
-use futures::{future};
-use futures_cpupool::CpuPool;
-use page::handler::{self, PageCache};
-use endpoint::{Endpoint, EndpointInfo, EndpointPath, Request, Response};
-use hyper::mime::Mime;
-
-#[derive(Clone)]
-pub struct Dapp {
-	pool: CpuPool,
-	path: PathBuf,
-	mime: Option<Mime>,
-	info: Option<EndpointInfo>,
-	cache: PageCache,
-}
-
-impl fmt::Debug for Dapp {
-	fn fmt(&self, fmt: &mut fmt::Formatter) -> fmt::Result {
-		fmt.debug_struct("Dapp")
-			.field("path", &self.path)
-			.field("mime", &self.mime)
-			.field("info", &self.info)
-			.field("cache", &self.cache)
-			.finish()
-	}
-}
-
-impl Dapp {
-	pub fn new(pool: CpuPool, path: PathBuf, info: EndpointInfo, cache: PageCache) -> Self {
-		Dapp {
-			pool,
-			path,
-			mime: None,
-			info: Some(info),
-			cache,
-		}
-	}
-
-	pub fn single_file(pool: CpuPool, path: PathBuf, mime: Mime, cache: PageCache) -> Self {
-		Dapp {
-			pool,
-			path,
-			mime: Some(mime),
-			info: None,
-			cache,
-		}
-	}
-
-	pub fn path(&self) -> PathBuf {
-		self.path.clone()
-	}
-
-	fn get_file(&self, path: &EndpointPath) -> Option<LocalFile> {
-		if let Some(ref mime) = self.mime {
-			return LocalFile::from_path(&self.path, mime.to_owned());
-		}
-
-		let mut file_path = self.path.to_owned();
-
-		if path.has_no_params() {
-			file_path.push("index.html");
-		} else {
-			for part in &path.app_params {
-				file_path.push(part);
-			}
-		}
-
-		let mime = mime_guess::guess_mime_type(&file_path);
-		LocalFile::from_path(&file_path, mime)
-	}
-
-	pub fn to_response(&self, path: &EndpointPath) -> Response {
-		let (reader, response) = handler::PageHandler {
-			file: self.get_file(path),
-			cache: self.cache,
-			allow_js_eval: self.info.as_ref().and_then(|x| x.allow_js_eval).unwrap_or(false),
-		}.into_response();
-
-		self.pool.spawn(reader).forget();
-
-		Box::new(future::ok(response))
-	}
-}
-
-impl Endpoint for Dapp {
-	fn info(&self) -> Option<&EndpointInfo> {
-		self.info.as_ref()
-	}
-
-	fn respond(&self, path: EndpointPath, _req: Request) -> Response {
-		self.to_response(&path)
-	}
-}
-
-struct LocalFile {
-	content_type: Mime,
-	file: fs::File,
-}
-
-impl LocalFile {
-	fn from_path<P: AsRef<Path>>(path: P, content_type: Mime) -> Option<Self> {
-		trace!(target: "dapps", "Local file: {:?}", path.as_ref());
-		// Check if file exists
-		fs::File::open(&path).ok().map(|file| {
-			LocalFile {
-				content_type,
-				file,
-			}
-		})
-	}
-}
-
-impl handler::DappFile for LocalFile {
-	type Reader = fs::File;
-
-	fn content_type(&self) -> &Mime {
-		&self.content_type
-	}
-
-	fn into_reader(self) -> Self::Reader {
-		self.file
-	}
-}

dapps/src/page/mod.rs

@@ -1,21 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-pub mod builtin;
-pub mod local;
-mod handler;
-
-pub use self::handler::PageCache;

dapps/src/proxypac.rs

@@ -1,61 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Serving ProxyPac file
-
-use apps::HOME_PAGE;
-use endpoint::{Endpoint, Request, Response, EndpointPath};
-use futures::future;
-use handlers::ContentHandler;
-use hyper::mime;
-
-pub struct ProxyPac {
-	dapps_domain: String,
-}
-
-impl ProxyPac {
-	pub fn boxed(dapps_domain: String) -> Box<Endpoint> {
-		Box::new(ProxyPac { dapps_domain })
-	}
-}
-
-impl Endpoint for ProxyPac {
-	fn respond(&self, path: EndpointPath, _req: Request) -> Response {
-		let ui = format!("{}:{}", path.host, path.port);
-
-		let content = format!(
-r#"
-function FindProxyForURL(url, host) {{
-	if (shExpMatch(host, "{0}.{1}"))
-	{{
-		return "PROXY {4}";
-	}}
-
-	if (shExpMatch(host, "*.{1}"))
-	{{
-		return "PROXY {2}:{3}";
-	}}
-
-	return "DIRECT";
-}}
-"#,
-		HOME_PAGE, self.dapps_domain, path.host, path.port, ui);
-
-		Box::new(future::ok(
-			ContentHandler::ok(content, mime::TEXT_JAVASCRIPT).into()
-		))
-	}
-}

dapps/src/router.rs

@@ -1,384 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Router implementation
-//! Dispatch requests to proper application.
-
-use std::sync::Arc;
-use std::collections::HashMap;
-
-use futures::future;
-use hyper::{self, header, Uri};
-use jsonrpc_http_server as http;
-
-use apps;
-use apps::fetcher::Fetcher;
-use endpoint::{self, Endpoint, EndpointPath};
-use Endpoints;
-use handlers;
-
-/// Special endpoints are accessible on every domain (every dapp)
-#[derive(Debug, PartialEq, Hash, Eq)]
-pub enum SpecialEndpoint {
-	Rpc,
-	Api,
-	Home,
-	None,
-}
-
-enum Response {
-	Some(endpoint::Response),
-	None(hyper::Request),
-}
-
-/// An endpoint router.
-/// Dispatches the request to particular Endpoint by requested uri/path.
-pub struct Router {
-	endpoints: Option<Endpoints>,
-	fetch: Arc<Fetcher>,
-	special: HashMap<SpecialEndpoint, Option<Box<Endpoint>>>,
-	dapps_domain: String,
-}
-
-impl Router {
-	fn resolve_request(&self, req: hyper::Request, refresh_dapps: bool) -> Response {
-		// Choose proper handler depending on path / domain
-		let endpoint = extract_endpoint(req.uri(), req.headers().get(), &self.dapps_domain);
-		let referer = extract_referer_endpoint(&req, &self.dapps_domain);
-		let is_get_request = *req.method() == hyper::Method::Get;
-		let is_head_request = *req.method() == hyper::Method::Head;
-		let has_dapp = |dapp: &str| self.endpoints
-			.as_ref()
-			.map_or(false, |endpoints| endpoints.endpoints.read().contains_key(dapp));
-
-		trace!(target: "dapps", "Routing request to {:?}. Details: {:?}", req.uri(), req);
-		debug!(target: "dapps", "Handling endpoint request: {:?}, referer: {:?}", endpoint, referer);
-
-		match (endpoint.0, endpoint.1, referer) {
-			// Handle invalid web requests that we can recover from
-			(ref path, SpecialEndpoint::None, Some(ref referer))
-				if referer.app_id == apps::WEB_PATH
-					&& has_dapp(apps::WEB_PATH)
-					&& !is_web_endpoint(path)
-				=>
-			{
-				let token = referer.app_params.get(0).map(String::as_str).unwrap_or("");
-				let requested = req.uri().path();
-				let query = req.uri().query().map_or_else(String::new, |query| format!("?{}", query));
-				let redirect_url = format!("/{}/{}{}{}", apps::WEB_PATH, token, requested, query);
-				trace!(target: "dapps", "Redirecting to correct web request: {:?}", redirect_url);
-				Response::Some(Box::new(future::ok(
-					handlers::Redirection::new(redirect_url).into()
-				)))
-			},
-			// First check special endpoints
-			(ref path, ref endpoint, _) if self.special.contains_key(endpoint) => {
-				trace!(target: "dapps", "Resolving to special endpoint.");
-				let special = self.special.get(endpoint).expect("special known to contain key; qed");
-				match *special {
-					Some(ref special) => Response::Some(special.respond(path.clone().unwrap_or_default(), req)),
-					None => Response::None(req),
-				}
-			},
-			// Then delegate to dapp
-			(Some(ref path), _, _) if has_dapp(&path.app_id) => {
-				trace!(target: "dapps", "Resolving to local/builtin dapp.");
-				Response::Some(self.endpoints
-					.as_ref()
-					.expect("endpoints known to be set; qed")
-					.endpoints
-					.read()
-					.get(&path.app_id)
-					.expect("endpoints known to contain key; qed")
-					.respond(path.clone(), req))
-			},
-			// Try to resolve and fetch the dapp
-			(Some(ref path), _, _) if self.fetch.contains(&path.app_id) => {
-				trace!(target: "dapps", "Resolving to fetchable content.");
-				Response::Some(self.fetch.respond(path.clone(), req))
-			},
-			// 404 for non-existent content (only if serving endpoints and not homepage)
-			(Some(ref path), _, _)
-				if (is_get_request || is_head_request)
-					&& self.endpoints.is_some()
-					&& path.app_id != apps::HOME_PAGE
-				=>
-			{
-				trace!(target: "dapps", "Resolving to 404.");
-				if refresh_dapps {
-					debug!(target: "dapps", "Refreshing dapps and re-trying.");
-					self.endpoints.as_ref().map(|endpoints| endpoints.refresh_local_dapps());
-					return self.resolve_request(req, false);
-				} else {
-					Response::Some(Box::new(future::ok(handlers::ContentHandler::error(
-						hyper::StatusCode::NotFound,
-						"404 Not Found",
-						"Requested content was not found.",
-						None,
-					).into())))
-				}
-			},
-			// Any other GET|HEAD requests to home page.
-			_ if (is_get_request || is_head_request) && self.special.contains_key(&SpecialEndpoint::Home) => {
-				trace!(target: "dapps", "Resolving to home page.");
-				let special = self.special.get(&SpecialEndpoint::Home).expect("special known to contain key; qed");
-				match *special {
-					Some(ref special) => {
-						let mut endpoint = EndpointPath::default();
-						endpoint.app_params = req.uri().path().split('/').map(str::to_owned).collect();
-						Response::Some(special.respond(endpoint, req))
-					},
-					None => Response::None(req),
-				}
-			},
-			// RPC by default
-			_ if self.special.contains_key(&SpecialEndpoint::Rpc) => {
-				trace!(target: "dapps", "Resolving to RPC call.");
-				Response::None(req)
-			},
-			// 404 otherwise
-			_ => {
-				Response::Some(Box::new(future::ok(handlers::ContentHandler::error(
-					hyper::StatusCode::NotFound,
-					"404 Not Found",
-					"Requested content was not found.",
-					None,
-				).into())))
-			},
-		}
-	}
-}
-
-impl http::RequestMiddleware for Router {
-	fn on_request(&self, req: hyper::Request) -> http::RequestMiddlewareAction {
-		let is_origin_set = req.headers().get::<header::Origin>().is_some();
-		let response = self.resolve_request(req, self.endpoints.is_some());
-		match response {
-			Response::Some(response) => http::RequestMiddlewareAction::Respond {
-				should_validate_hosts: true,
-				response,
-			},
-			Response::None(request) => http::RequestMiddlewareAction::Proceed {
-				should_continue_on_invalid_cors: !is_origin_set,
-				request,
-			},
-		}
-	}
-}
-
-impl Router {
-	pub fn new(
-		content_fetcher: Arc<Fetcher>,
-		endpoints: Option<Endpoints>,
-		special: HashMap<SpecialEndpoint, Option<Box<Endpoint>>>,
-		dapps_domain: String,
-	) -> Self {
-		Router {
-			endpoints: endpoints,
-			fetch: content_fetcher,
-			special: special,
-			dapps_domain: format!(".{}", dapps_domain),
-		}
-	}
-}
-
-fn is_web_endpoint(path: &Option<EndpointPath>) -> bool {
-	match *path {
-		Some(ref path) if path.app_id == apps::WEB_PATH => true,
-		_ => false,
-	}
-}
-
-fn extract_referer_endpoint(req: &hyper::Request, dapps_domain: &str) -> Option<EndpointPath> {
-	let referer = req.headers().get::<header::Referer>();
-
-	let url = referer.and_then(|referer| referer.parse().ok());
-	url.and_then(|url| {
-		extract_url_referer_endpoint(&url, dapps_domain).or_else(|| {
-			extract_endpoint(&url, None, dapps_domain).0
-		})
-	})
-}
-
-fn extract_url_referer_endpoint(url: &Uri, dapps_domain: &str) -> Option<EndpointPath> {
-	let query = url.query();
-	match query {
-		Some(query) if query.starts_with(apps::URL_REFERER) => {
-			let scheme = url.scheme().unwrap_or("http");
-			let host = url.host().unwrap_or("unknown");
-			let port = default_port(url, None);
-			let referer_url = format!("{}://{}:{}/{}", scheme, host, port, &query[apps::URL_REFERER.len()..]);
-			debug!(target: "dapps", "Recovering referer from query parameter: {}", referer_url);
-
-			if let Some(referer_url) = referer_url.parse().ok() {
-				extract_endpoint(&referer_url, None, dapps_domain).0
-			} else {
-				None
-			}
-		},
-		_ => None,
-	}
-}
-
-fn extract_endpoint(url: &Uri, extra_host: Option<&header::Host>, dapps_domain: &str) -> (Option<EndpointPath>, SpecialEndpoint) {
-	fn special_endpoint(path: &[&str]) -> SpecialEndpoint {
-		if path.len() <= 1 {
-			return SpecialEndpoint::None;
-		}
-
-		match path[0].as_ref() {
-			apps::RPC_PATH => SpecialEndpoint::Rpc,
-			apps::API_PATH => SpecialEndpoint::Api,
-			apps::HOME_PAGE => SpecialEndpoint::Home,
-			_ => SpecialEndpoint::None,
-		}
-	}
-
-	let port = default_port(url, extra_host.as_ref().and_then(|h| h.port()));
-	let host = url.host().or_else(|| extra_host.as_ref().map(|h| h.hostname()));
-	let query = url.query().map(str::to_owned);
-	let mut path_segments = url.path().split('/').skip(1).collect::<Vec<_>>();
-	trace!(
-		target: "dapps",
-		"Extracting endpoint from: {:?} (dapps: {}). Got host {:?}:{} with path {:?}",
-		url, dapps_domain, host, port, path_segments
-	);
-	match host {
-		Some(host) if host.ends_with(dapps_domain) => {
-			let id = &host[0..(host.len() - dapps_domain.len())];
-			let special = special_endpoint(&path_segments);
-
-			// remove special endpoint id from params
-			if special != SpecialEndpoint::None {
-				path_segments.remove(0);
-			}
-
-			let (app_id, app_params) = if let Some(split) = id.rfind('.') {
-				let (params, id) = id.split_at(split);
-				path_segments.insert(0, params);
-				(id[1..].to_owned(), path_segments)
-			} else {
-				(id.to_owned(), path_segments)
-			};
-
-			(Some(EndpointPath {
-				app_id,
-				app_params: app_params.into_iter().map(Into::into).collect(),
-				query,
-				host: host.to_owned(),
-				port,
-				using_dapps_domains: true,
-			}), special)
-		},
-		Some(host) if path_segments.len() > 1 => {
-			let special = special_endpoint(&path_segments);
-			let id = path_segments.remove(0);
-			(Some(EndpointPath {
-				app_id: id.to_owned(),
-				app_params: path_segments.into_iter().map(Into::into).collect(),
-				query,
-				host: host.to_owned(),
-				port,
-				using_dapps_domains: false,
-			}), special)
-		},
-		_ => (None, special_endpoint(&path_segments)),
-	}
-}
-
-fn default_port(url: &Uri, extra_port: Option<u16>) -> u16 {
-	let scheme = url.scheme().unwrap_or("http");
-	url.port().or(extra_port).unwrap_or_else(|| match scheme {
-		"http" => 80,
-		"https" => 443,
-		_ => 80,
-	})
-}
-
-#[cfg(test)]
-mod tests {
-	use super::{SpecialEndpoint, EndpointPath, extract_endpoint};
-
-	#[test]
-	fn should_extract_endpoint() {
-		let dapps_domain = ".web3.site";
-
-		// With path prefix
-		assert_eq!(
-			extract_endpoint(&"http://localhost:8080/status/index.html?q=1".parse().unwrap(), None, dapps_domain),
-			(Some(EndpointPath {
-				app_id: "status".to_owned(),
-				app_params: vec!["index.html".to_owned()],
-				query: Some("q=1".into()),
-				host: "localhost".to_owned(),
-				port: 8080,
-				using_dapps_domains: false,
-			}), SpecialEndpoint::None)
-		);
-
-		// With path prefix
-		assert_eq!(
-			extract_endpoint(&"http://localhost:8080/rpc/".parse().unwrap(), None, dapps_domain),
-			(Some(EndpointPath {
-				app_id: "rpc".to_owned(),
-				app_params: vec!["".to_owned()],
-				query: None,
-				host: "localhost".to_owned(),
-				port: 8080,
-				using_dapps_domains: false,
-			}), SpecialEndpoint::Rpc)
-		);
-
-		// By Subdomain
-		assert_eq!(
-			extract_endpoint(&"http://status.web3.site/test.html".parse().unwrap(), None, dapps_domain),
-			(Some(EndpointPath {
-				app_id: "status".to_owned(),
-				app_params: vec!["test.html".to_owned()],
-				query: None,
-				host: "status.web3.site".to_owned(),
-				port: 80,
-				using_dapps_domains: true,
-			}), SpecialEndpoint::None)
-		);
-
-		// RPC by subdomain
-		assert_eq!(
-			extract_endpoint(&"http://my.status.web3.site/rpc/".parse().unwrap(), None, dapps_domain),
-			(Some(EndpointPath {
-				app_id: "status".to_owned(),
-				app_params: vec!["my".into(), "".into()],
-				query: None,
-				host: "my.status.web3.site".to_owned(),
-				port: 80,
-				using_dapps_domains: true,
-			}), SpecialEndpoint::Rpc)
-		);
-
-		// API by subdomain
-		assert_eq!(
-			extract_endpoint(&"http://my.status.web3.site/api/".parse().unwrap(), None, dapps_domain),
-			(Some(EndpointPath {
-				app_id: "status".to_owned(),
-				app_params: vec!["my".into(), "".into()],
-				query: None,
-				host: "my.status.web3.site".to_owned(),
-				port: 80,
-				using_dapps_domains: true,
-			}), SpecialEndpoint::Api)
-		);
-	}
-}

dapps/src/tests/api.rs

@@ -1,85 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use tests::helpers::{serve, serve_with_registrar, request, assert_security_headers};
-
-#[test]
-fn should_return_error() {
-	// given
-	let server = serve();
-
-	// when
-	let response = request(server,
-		"\
-			GET /api/empty HTTP/1.1\r\n\
-			Host: 127.0.0.1:8080\r\n\
-			Connection: close\r\n\
-			\r\n\
-			{}
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 404 Not Found");
-	response.assert_header("Content-Type", "application/json");
-	assert_eq!(response.body, format!("58\n{}\n0\n\n", r#"{"code":"404","title":"Not Found","detail":"Resource you requested has not been found."}"#));
-	assert_security_headers(&response.headers);
-}
-
-#[test]
-fn should_handle_ping() {
-	// given
-	let server = serve();
-
-	// when
-	let response = request(server,
-		"\
-			POST /api/ping HTTP/1.1\r\n\
-			Host: home.parity\r\n\
-			Content-Type: application/json\r\n\
-			Connection: close\r\n\
-			\r\n\
-			{}
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 200 OK");
-	response.assert_header("Content-Type", "application/json");
-	assert_eq!(response.body, "0\n\n".to_owned());
-	assert_security_headers(&response.headers);
-}
-
-#[test]
-fn should_try_to_resolve_dapp() {
-	// given
-	let (server, registrar) = serve_with_registrar();
-
-	// when
-	let response = request(server,
-		"\
-			GET /api/content/1472a9e190620cdf6b31f383373e45efcfe869a820c91f9ccd7eb9fb45e4985d HTTP/1.1\r\n\
-			Host: home.parity\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 404 Not Found");
-	assert_eq!(registrar.calls.lock().len(), 2);
-	assert_security_headers(&response.headers);
-}

dapps/src/tests/fetch.rs

@@ -1,544 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use devtools::http_client;
-use rustc_hex::FromHex;
-use tests::helpers::{
-	serve_with_registrar, serve_with_registrar_and_sync, serve_with_fetch,
-	serve_with_registrar_and_fetch,
-	request, assert_security_headers
-};
-
-#[test]
-fn should_resolve_dapp() {
-	// given
-	let (server, registrar) = serve_with_registrar();
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: 1472a9e190620cdf6b31f383373e45efcfe869a820c91f9ccd7eb9fb45e4985d.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 404 Not Found");
-	assert_eq!(registrar.calls.lock().len(), 4);
-	assert_security_headers(&response.headers);
-}
-
-#[test]
-fn should_return_503_when_syncing_but_should_make_the_calls() {
-	// given
-	let (server, registrar) = serve_with_registrar_and_sync();
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: 1472a9e190620cdf6b31f383373e45efcfe869a820c91f9ccd7eb9fb45e4985d.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 503 Service Unavailable");
-	assert_eq!(registrar.calls.lock().len(), 2);
-	assert_security_headers(&response.headers);
-}
-
-const GAVCOIN_DAPP: &'static str = "00000000000000000000000000000000000000000000000000000000000000609faf32e1e3845e237cc6efd27187cee13b3b99db000000000000000000000000000000000000000000000000d8bd350823e28ff75e74a34215faefdc8a52fd8e00000000000000000000000000000000000000000000000000000000000000116761766f66796f726b2f676176636f696e000000000000000000000000000000";
-const GAVCOIN_ICON: &'static str = "00000000000000000000000000000000000000000000000000000000000000600000000000000000000000000000000000000000000000000000000000000000000000000000000000000000d8bd350823e28ff75e74a34215faefdc8a52fd8e000000000000000000000000000000000000000000000000000000000000007768747470733a2f2f7261772e67697468756275736572636f6e74656e742e636f6d2f657468636f72652f646170702d6173736574732f623838653938336162616131613661363334356238643934343863313562313137646462353430652f746f6b656e732f676176636f696e2d36347836342e706e67000000000000000000";
-
-#[test]
-fn should_return_502_on_hash_mismatch() {
-	// given
-	let (server, fetch, registrar) = serve_with_registrar_and_fetch();
-	let gavcoin = GAVCOIN_DAPP.from_hex().unwrap();
-	registrar.set_result(
-		"94f093625c06887d94d9fee0d5f9cc4aaa46f33d24d1c7e4b5237e7c37d547dd".parse().unwrap(),
-		Ok(gavcoin.clone())
-	);
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: 94f093625c06887d94d9fee0d5f9cc4aaa46f33d24d1c7e4b5237e7c37d547dd.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	assert_eq!(registrar.calls.lock().len(), 4);
-
-	fetch.assert_requested("https://codeload.github.com/gavofyork/gavcoin/zip/9faf32e1e3845e237cc6efd27187cee13b3b99db");
-	fetch.assert_no_more_requests();
-
-	response.assert_status("HTTP/1.1 502 Bad Gateway");
-	assert!(response.body.contains("HashMismatch"), "Expected hash mismatch response, got: {:?}", response.body);
-	assert_security_headers(&response.headers);
-}
-
-#[test]
-fn should_return_error_for_invalid_dapp_zip() {
-	// given
-	let (server, fetch, registrar) = serve_with_registrar_and_fetch();
-	let gavcoin = GAVCOIN_DAPP.from_hex().unwrap();
-	registrar.set_result(
-		"2be00befcf008bc0e7d9cdefc194db9c75352e8632f48498b5a6bfce9f02c88e".parse().unwrap(),
-		Ok(gavcoin.clone())
-	);
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: 2be00befcf008bc0e7d9cdefc194db9c75352e8632f48498b5a6bfce9f02c88e.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	assert_eq!(registrar.calls.lock().len(), 4);
-
-	fetch.assert_requested("https://codeload.github.com/gavofyork/gavcoin/zip/9faf32e1e3845e237cc6efd27187cee13b3b99db");
-	fetch.assert_no_more_requests();
-
-	response.assert_status("HTTP/1.1 502 Bad Gateway");
-	assert!(response.body.contains("InvalidArchive"), "Expected invalid zip response, got: {:?}", response.body);
-	assert_security_headers(&response.headers);
-}
-
-#[test]
-fn should_return_fetched_dapp_content() {
-	// given
-	let (server, fetch, registrar) = serve_with_registrar_and_fetch();
-	let gavcoin = GAVCOIN_DAPP.from_hex().unwrap();
-	registrar.set_result(
-		"9c94e154dab8acf859b30ee80fc828fb1d38359d938751b65db71d460588d82a".parse().unwrap(),
-		Ok(gavcoin.clone())
-	);
-	fetch.set_response(include_bytes!("../../res/gavcoin.zip"));
-
-	// when
-	let response1 = http_client::request(server.addr(),
-		"\
-			GET /index.html HTTP/1.1\r\n\
-			Host: 9c94e154dab8acf859b30ee80fc828fb1d38359d938751b65db71d460588d82a.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-	let response2 = http_client::request(server.addr(),
-		"\
-			GET /manifest.json HTTP/1.1\r\n\
-			Host: 9c94e154dab8acf859b30ee80fc828fb1d38359d938751b65db71d460588d82a.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	assert_eq!(registrar.calls.lock().len(), 4);
-
-	fetch.assert_requested("https://codeload.github.com/gavofyork/gavcoin/zip/9faf32e1e3845e237cc6efd27187cee13b3b99db");
-	fetch.assert_no_more_requests();
-
-	response1.assert_status("HTTP/1.1 200 OK");
-	assert_security_headers(&response1.headers);
-	assert!(
-		response1.body.contains(r#"18
-<h1>Hello Gavcoin!</h1>
-
-0
-
-"#),
-		"Expected Gavcoin body: {}",
-		response1.body
-	);
-
-	response2.assert_status("HTTP/1.1 200 OK");
-	assert_security_headers(&response2.headers);
-	assert_eq!(
-		response2.body,
-		r#"EA
-{
-  "id": "9c94e154dab8acf859b30ee80fc828fb1d38359d938751b65db71d460588d82a",
-  "name": "Gavcoin",
-  "description": "Gavcoin",
-  "version": "1.0.0",
-  "author": "",
-  "iconUrl": "icon.png",
-  "localUrl": null,
-  "allowJsEval": false
-}
-0
-
-"#
-	);
-}
-
-#[test]
-fn should_return_fetched_content() {
-	// given
-	let (server, fetch, registrar) = serve_with_registrar_and_fetch();
-	let gavcoin = GAVCOIN_ICON.from_hex().unwrap();
-	registrar.set_result(
-		"2be00befcf008bc0e7d9cdefc194db9c75352e8632f48498b5a6bfce9f02c88e".parse().unwrap(),
-		Ok(gavcoin.clone())
-	);
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: 2be00befcf008bc0e7d9cdefc194db9c75352e8632f48498b5a6bfce9f02c88e.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	assert_eq!(registrar.calls.lock().len(), 4);
-
-	fetch.assert_requested("https://raw.githubusercontent.com/ethcore/dapp-assets/b88e983abaa1a6a6345b8d9448c15b117ddb540e/tokens/gavcoin-64x64.png");
-	fetch.assert_no_more_requests();
-
-	response.assert_status("HTTP/1.1 200 OK");
-	response.assert_security_headers_present(None);
-}
-
-#[test]
-fn should_cache_content() {
-	// given
-	let (server, fetch, registrar) = serve_with_registrar_and_fetch();
-	let gavcoin = GAVCOIN_ICON.from_hex().unwrap();
-	registrar.set_result(
-		"2be00befcf008bc0e7d9cdefc194db9c75352e8632f48498b5a6bfce9f02c88e".parse().unwrap(),
-		Ok(gavcoin.clone())
-	);
-	let request_str = "\
-		GET / HTTP/1.1\r\n\
-		Host: 2be00befcf008bc0e7d9cdefc194db9c75352e8632f48498b5a6bfce9f02c88e.web3.site\r\n\
-		Connection: close\r\n\
-		\r\n\
-	";
-
-	let response = http_client::request(server.addr(), request_str);
-	fetch.assert_requested("https://raw.githubusercontent.com/ethcore/dapp-assets/b88e983abaa1a6a6345b8d9448c15b117ddb540e/tokens/gavcoin-64x64.png");
-	fetch.assert_no_more_requests();
-	response.assert_status("HTTP/1.1 200 OK");
-
-	// when
-	let response = http_client::request(server.addr(), request_str);
-
-	// then
-	fetch.assert_no_more_requests();
-	response.assert_status("HTTP/1.1 200 OK");
-}
-
-#[test]
-fn should_not_request_content_twice() {
-	use std::thread;
-
-	// given
-	let (server, fetch, registrar) = serve_with_registrar_and_fetch();
-	let gavcoin = GAVCOIN_ICON.from_hex().unwrap();
-	registrar.set_result(
-		"2be00befcf008bc0e7d9cdefc194db9c75352e8632f48498b5a6bfce9f02c88e".parse().unwrap(),
-		Ok(gavcoin.clone())
-	);
-	let request_str = "\
-		GET / HTTP/1.1\r\n\
-		Host: 2be00befcf008bc0e7d9cdefc194db9c75352e8632f48498b5a6bfce9f02c88e.web3.site\r\n\
-		Connection: close\r\n\
-		\r\n\
-	";
-	let fire_request = || {
-		let addr = server.addr().to_owned();
-		let req = request_str.to_owned();
-		thread::spawn(move || {
-			http_client::request(&addr, &req)
-		})
-	};
-	let control = fetch.manual();
-
-	// when
-
-	// Fire two requests at the same time
-	let r1 = fire_request();
-	let r2 = fire_request();
-
-	// wait for single request in fetch, the second one should go into waiting state.
-	control.wait_for_requests(1);
-	control.respond();
-
-	let response1 = r1.join().unwrap();
-	let response2 = r2.join().unwrap();
-
-	// then
-	fetch.assert_requested("https://raw.githubusercontent.com/ethcore/dapp-assets/b88e983abaa1a6a6345b8d9448c15b117ddb540e/tokens/gavcoin-64x64.png");
-	fetch.assert_no_more_requests();
-	response1.assert_status("HTTP/1.1 200 OK");
-	response2.assert_status("HTTP/1.1 200 OK");
-}
-
-#[test]
-fn should_encode_and_decode_base32() {
-	use base32;
-
-	let encoded = base32::encode(base32::Alphabet::Crockford, "token+https://parity.io".as_bytes());
-	assert_eq!("EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY", &encoded);
-
-	let data = base32::decode(base32::Alphabet::Crockford, "EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY").unwrap();
-	assert_eq!("token+https://parity.io", &String::from_utf8(data).unwrap());
-}
-
-#[test]
-fn should_stream_web_content() {
-	// given
-	let (server, fetch) = serve_with_fetch("token", "https://parity.io");
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY.web.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 200 OK");
-	assert_security_headers(&response.headers);
-
-	fetch.assert_requested("https://parity.io/");
-	fetch.assert_no_more_requests();
-}
-
-#[test]
-fn should_support_base32_encoded_web_urls() {
-	// given
-	let (server, fetch) = serve_with_fetch("token", "https://parity.io");
-
-	// when
-	let response = request(server,
-		"\
-			GET /styles.css?test=123 HTTP/1.1\r\n\
-			Host: EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY.web.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 200 OK");
-	assert_security_headers(&response.headers);
-
-	fetch.assert_requested("https://parity.io/styles.css?test=123");
-	fetch.assert_no_more_requests();
-}
-
-#[test]
-fn should_correctly_handle_long_label_when_splitted() {
-	// given
-	let (server, fetch) = serve_with_fetch("xolrg9fePeQyKLnL", "https://contribution.melonport.com");
-
-	// when
-	let response = request(server,
-		"\
-			GET /styles.css?test=123 HTTP/1.1\r\n\
-			Host: f1qprwk775k6am35a5wmpk3e9gnpgx3me1sk.mbsfcdqpwx3jd5h7ax39dxq2wvb5dhqpww3fe9t2wrvfdm.web.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 200 OK");
-	assert_security_headers(&response.headers);
-
-	fetch.assert_requested("https://contribution.melonport.com/styles.css?test=123");
-	fetch.assert_no_more_requests();
-}
-
-#[test]
-fn should_support_base32_encoded_web_urls_as_path() {
-	// given
-	let (server, fetch) = serve_with_fetch("token", "https://parity.io");
-
-	// when
-	let response = request(server,
-		"\
-			GET /web/EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY/styles.css?test=123 HTTP/1.1\r\n\
-			Host: localhost:8080\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 200 OK");
-	assert_security_headers(&response.headers);
-
-	fetch.assert_requested("https://parity.io/styles.css?test=123");
-	fetch.assert_no_more_requests();
-}
-
-#[test]
-fn should_return_error_on_non_whitelisted_domain() {
-	// given
-	let (server, fetch) = serve_with_fetch("token", "https://ethcore.io");
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY.web.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 400 Bad Request");
-	assert_security_headers(&response.headers);
-
-	fetch.assert_no_more_requests();
-}
-
-#[test]
-fn should_return_error_on_invalid_token() {
-	// given
-	let (server, fetch) = serve_with_fetch("test", "https://parity.io");
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY.web.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 400 Bad Request");
-	assert_security_headers(&response.headers);
-
-	fetch.assert_no_more_requests();
-}
-
-#[test]
-fn should_return_error_on_invalid_protocol() {
-	// given
-	let (server, fetch) = serve_with_fetch("token", "ftp://parity.io");
-
-	// when
-	let response = request(server,
-		"\
-			GET /web/token/ftp/parity.io/ HTTP/1.1\r\n\
-			Host: localhost:8080\r\n\
-			Connection: close\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 400 Bad Request");
-	assert_security_headers(&response.headers);
-
-	fetch.assert_no_more_requests();
-}
-
-#[test]
-fn should_disallow_non_get_requests() {
-	// given
-	let (server, fetch) = serve_with_fetch("token", "https://parity.io");
-
-	// when
-	let response = request(server,
-		"\
-			POST / HTTP/1.1\r\n\
-			Host: EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY.web.web3.site\r\n\
-			Content-Type: application/json\r\n\
-			Connection: close\r\n\
-			\r\n\
-			123\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 405 Method Not Allowed");
-	assert_security_headers(&response.headers);
-
-	fetch.assert_no_more_requests();
-}
-
-#[test]
-fn should_fix_absolute_requests_based_on_referer() {
-	// given
-	let (server, fetch) = serve_with_fetch("token", "https://parity.io");
-
-	// when
-	let response = request(server,
-		"\
-			GET /styles.css HTTP/1.1\r\n\
-			Host: localhost:8080\r\n\
-			Connection: close\r\n\
-			Referer: http://localhost:8080/web/EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY/\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 302 Found");
-	response.assert_header("Location", "/web/EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY/styles.css");
-
-	fetch.assert_no_more_requests();
-}
-
-#[test]
-fn should_fix_absolute_requests_based_on_referer_in_url() {
-	// given
-	let (server, fetch) = serve_with_fetch("token", "https://parity.io");
-
-	// when
-	let response = request(server,
-		"\
-			GET /styles.css HTTP/1.1\r\n\
-			Host: localhost:8080\r\n\
-			Connection: close\r\n\
-			Referer: http://localhost:8080/?__referer=web/EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY/\r\n\
-			\r\n\
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 302 Found");
-	response.assert_header("Location", "/web/EHQPPSBE5DM78X3GECX2YBVGC5S6JX3S5SMPY/styles.css");
-
-	fetch.assert_no_more_requests();
-}

dapps/src/tests/helpers/fetch.rs

@@ -1,134 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use std::{thread, time};
-use std::sync::{atomic, mpsc, Arc};
-use parking_lot::Mutex;
-use hyper;
-
-use futures::{self, future, Future};
-use fetch::{self, Fetch, Url, Request, Abort};
-
-pub struct FetchControl {
-	sender: mpsc::Sender<()>,
-	fetch: FakeFetch,
-}
-
-impl FetchControl {
-	pub fn respond(self) {
-		self.sender.send(())
-			.expect("Fetch cannot be finished without sending a response at least once.");
-	}
-
-	pub fn wait_for_requests(&self, len: usize) {
-		const MAX_TIMEOUT: time::Duration = time::Duration::from_millis(5000);
-		const ATTEMPTS: u32 = 10;
-		let mut attempts_left = ATTEMPTS;
-		loop {
-			let current = self.fetch.requested.lock().len();
-
-			if current == len {
-				break;
-			} else if attempts_left == 0 {
-				panic!(
-					"Timeout reached when waiting for pending requests. Expected: {}, current: {}",
-					len, current
-				);
-			} else {
-				attempts_left -= 1;
-				// Should we handle spurious timeouts better?
-				thread::park_timeout(MAX_TIMEOUT / ATTEMPTS);
-			}
-		}
-	}
-}
-
-#[derive(Clone, Default)]
-pub struct FakeFetch {
-	manual: Arc<Mutex<Option<mpsc::Receiver<()>>>>,
-	response: Arc<Mutex<Option<&'static [u8]>>>,
-	asserted: Arc<atomic::AtomicUsize>,
-	requested: Arc<Mutex<Vec<String>>>,
-}
-
-impl FakeFetch {
-	pub fn set_response(&self, data: &'static [u8]) {
-		*self.response.lock() = Some(data);
-	}
-
-	pub fn manual(&self) -> FetchControl {
-		assert!(self.manual.lock().is_none(), "Only one manual control may be active.");
-		let (tx, rx) = mpsc::channel();
-		*self.manual.lock() = Some(rx);
-
-		FetchControl {
-			sender: tx,
-			fetch: self.clone(),
-		}
-	}
-
-	pub fn assert_requested(&self, url: &str) {
-		let requests = self.requested.lock();
-		let idx = self.asserted.fetch_add(1, atomic::Ordering::SeqCst);
-
-		assert_eq!(requests.get(idx), Some(&url.to_owned()), "Expected fetch from specific URL.");
-	}
-
-	pub fn assert_no_more_requests(&self) {
-		let requests = self.requested.lock();
-		let len = self.asserted.load(atomic::Ordering::SeqCst);
-		assert_eq!(requests.len(), len, "Didn't expect any more requests, got: {:?}", &requests[len..]);
-	}
-}
-
-impl Fetch for FakeFetch {
-	type Result = Box<Future<Item = fetch::Response, Error = fetch::Error> + Send>;
-
-	fn fetch(&self, request: Request, abort: fetch::Abort) -> Self::Result {
-		let u = request.url().clone();
-		self.requested.lock().push(u.as_str().into());
-		let manual = self.manual.clone();
-		let response = self.response.clone();
-
-		let (tx, rx) = futures::oneshot();
-		thread::spawn(move || {
-			if let Some(rx) = manual.lock().take() {
-				// wait for manual resume
-				let _ = rx.recv();
-			}
-			let data = response.lock().take().unwrap_or(b"Some content");
-			tx.send(fetch::Response::new(u, hyper::Response::new().with_body(data), abort)).unwrap();
-		});
-
-		Box::new(rx.map_err(|_| fetch::Error::Aborted))
-	}
-
-	fn get(&self, url: &str, abort: Abort) -> Self::Result {
-		let url: Url = match url.parse() {
-			Ok(u) => u,
-			Err(e) => return Box::new(future::err(e.into()))
-		};
-		self.fetch(Request::get(url), abort)
-	}
-
-	fn post(&self, url: &str, abort: Abort) -> Self::Result {
-		let url: Url = match url.parse() {
-			Ok(u) => u,
-			Err(e) => return Box::new(future::err(e.into()))
-		};
-		self.fetch(Request::post(url), abort)
-	}
-}

dapps/src/tests/helpers/mod.rs

@@ -1,252 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use std::{env, io, str};
-use std::net::SocketAddr;
-use std::path::{Path, PathBuf};
-use std::sync::Arc;
-use env_logger::LogBuilder;
-use jsonrpc_core::IoHandler;
-use jsonrpc_http_server::{self as http, Host, DomainsValidation};
-use parity_reactor::Remote;
-
-use devtools::http_client;
-use registrar::{RegistrarClient, Asynchronous};
-use fetch::{Fetch, Client as FetchClient};
-use node_health::{NodeHealth, TimeChecker, CpuPool};
-
-use {Middleware, SyncStatus, WebProxyTokens};
-
-mod registrar;
-mod fetch;
-
-use self::registrar::FakeRegistrar;
-use self::fetch::FakeFetch;
-
-#[derive(Debug)]
-struct FakeSync(bool);
-impl SyncStatus for FakeSync {
-	fn is_major_importing(&self) -> bool { self.0 }
-	fn peers(&self) -> (usize, usize) { (0, 5) }
-}
-
-fn init_logger() {
-	// Initialize logger
-	if let Ok(log) = env::var("RUST_LOG") {
-		let mut builder = LogBuilder::new();
-		builder.parse(&log);
-		let _ = builder.init();	// ignore errors since ./test.sh will call this multiple times.
-	}
-}
-
-pub fn init_server<F, B>(process: F, io: IoHandler) -> (Server, Arc<FakeRegistrar>) where
-	F: FnOnce(ServerBuilder) -> ServerBuilder<B>,
-	B: Fetch,
-{
-	init_logger();
-	let registrar = Arc::new(FakeRegistrar::new());
-	let mut dapps_path = env::temp_dir();
-	dapps_path.push("non-existent-dir-to-prevent-fs-files-from-loading");
-
-	let builder = ServerBuilder::new(FetchClient::new().unwrap(), &dapps_path, registrar.clone());
-	let server = process(builder).start_unsecured_http(&"127.0.0.1:0".parse().unwrap(), io).unwrap();
-	(
-		server,
-		registrar,
-	)
-}
-
-pub fn serve_with_rpc(io: IoHandler) -> Server {
-	init_server(|builder| builder, io).0
-}
-
-pub fn serve_hosts(hosts: Option<Vec<String>>) -> Server {
-	let hosts = hosts.map(|hosts| hosts.into_iter().map(Into::into).collect());
-	init_server(|mut builder| {
-		builder.allowed_hosts = hosts.into();
-		builder
-	}, Default::default()).0
-}
-
-pub fn serve_with_registrar() -> (Server, Arc<FakeRegistrar>) {
-	init_server(|builder| builder, Default::default())
-}
-
-pub fn serve_with_registrar_and_sync() -> (Server, Arc<FakeRegistrar>) {
-	init_server(|mut builder| {
-		builder.sync_status = Arc::new(FakeSync(true));
-		builder
-	}, Default::default())
-}
-
-pub fn serve_with_registrar_and_fetch() -> (Server, FakeFetch, Arc<FakeRegistrar>) {
-	let fetch = FakeFetch::default();
-	let f = fetch.clone();
-	let (server, reg) = init_server(move |builder| {
-		builder.fetch(f.clone())
-	}, Default::default());
-
-	(server, fetch, reg)
-}
-
-pub fn serve_with_fetch(web_token: &'static str, domain: &'static str) -> (Server, FakeFetch) {
-	let fetch = FakeFetch::default();
-	let f = fetch.clone();
-	let (server, _) = init_server(move |mut builder| {
-		builder.web_proxy_tokens = Arc::new(move |token| {
-			if &token == web_token { Some(domain.into()) } else { None }
-		});
-		builder.fetch(f.clone())
-	}, Default::default());
-
-	(server, fetch)
-}
-
-pub fn serve() -> Server {
-	init_server(|builder| builder, Default::default()).0
-}
-
-pub fn request(server: Server, request: &str) -> http_client::Response {
-	http_client::request(server.addr(), request)
-}
-
-pub fn assert_security_headers(headers: &[String]) {
-	http_client::assert_security_headers_present(headers, None)
-}
-
-/// Webapps HTTP+RPC server build.
-pub struct ServerBuilder<T: Fetch = FetchClient> {
-	dapps_path: PathBuf,
-	registrar: Arc<RegistrarClient<Call=Asynchronous>>,
-	sync_status: Arc<SyncStatus>,
-	web_proxy_tokens: Arc<WebProxyTokens>,
-	allowed_hosts: DomainsValidation<Host>,
-	fetch: T,
-}
-
-impl ServerBuilder {
-	/// Construct new dapps server
-	pub fn new<P: AsRef<Path>>(fetch: FetchClient, dapps_path: P, registrar: Arc<RegistrarClient<Call=Asynchronous>>) -> Self {
-		ServerBuilder {
-			dapps_path: dapps_path.as_ref().to_owned(),
-			registrar: registrar,
-			sync_status: Arc::new(FakeSync(false)),
-			web_proxy_tokens: Arc::new(|_| None),
-			allowed_hosts: DomainsValidation::Disabled,
-			fetch: fetch,
-		}
-	}
-}
-
-impl<T: Fetch> ServerBuilder<T> {
-	/// Set a fetch client to use.
-	pub fn fetch<X: Fetch>(self, fetch: X) -> ServerBuilder<X> {
-		ServerBuilder {
-			dapps_path: self.dapps_path,
-			registrar: self.registrar,
-			sync_status: self.sync_status,
-			web_proxy_tokens: self.web_proxy_tokens,
-			allowed_hosts: self.allowed_hosts,
-			fetch: fetch,
-		}
-	}
-
-	/// Asynchronously start server with no authentication,
-	/// returns result with `Server` handle on success or an error.
-	pub fn start_unsecured_http(self, addr: &SocketAddr, io: IoHandler) -> io::Result<Server> {
-		Server::start_http(
-			addr,
-			io,
-			self.allowed_hosts,
-			self.dapps_path,
-			vec![],
-			self.registrar,
-			self.sync_status,
-			self.web_proxy_tokens,
-			Remote::new_sync(),
-			self.fetch,
-		)
-	}
-}
-
-const DAPPS_DOMAIN: &'static str = "web3.site";
-
-/// Webapps HTTP server.
-pub struct Server {
-	server: Option<http::Server>,
-}
-
-impl Server {
-	fn start_http<F: Fetch>(
-		addr: &SocketAddr,
-		io: IoHandler,
-		allowed_hosts: DomainsValidation<Host>,
-		dapps_path: PathBuf,
-		extra_dapps: Vec<PathBuf>,
-		registrar: Arc<RegistrarClient<Call=Asynchronous>>,
-		sync_status: Arc<SyncStatus>,
-		web_proxy_tokens: Arc<WebProxyTokens>,
-		remote: Remote,
-		fetch: F,
-	) -> io::Result<Server> {
-		let health = NodeHealth::new(
-			sync_status.clone(),
-			TimeChecker::new::<String>(&[], CpuPool::new(1)),
-			remote.clone(),
-		);
-		let pool = ::futures_cpupool::CpuPool::new(1);
-		let middleware =
-			Middleware::dapps(
-				pool,
-				health,
-				dapps_path,
-				extra_dapps,
-				DAPPS_DOMAIN.into(),
-				registrar,
-				sync_status,
-				web_proxy_tokens,
-				fetch,
-			);
-
-		let mut allowed_hosts: Option<Vec<Host>> = allowed_hosts.into();
-		allowed_hosts.as_mut().map(|hosts| {
-			hosts.push(format!("http://*.{}:*", DAPPS_DOMAIN).into());
-			hosts.push(format!("http://*.{}", DAPPS_DOMAIN).into());
-		});
-
-		http::ServerBuilder::new(io)
-			.request_middleware(middleware)
-			.allowed_hosts(allowed_hosts.into())
-			.cors(http::DomainsValidation::Disabled)
-			.start_http(addr)
-			.map(|server| Server {
-				server: Some(server),
-			})
-	}
-
-	/// Returns address that this server is bound to.
-	pub fn addr(&self) -> &SocketAddr {
-		self.server.as_ref()
-			.expect("server is always Some at the start; it's consumed only when object is dropped; qed")
-			.address()
-	}
-}
-
-impl Drop for Server {
-	fn drop(&mut self) {
-		self.server.take().unwrap().close()
-	}
-}

dapps/src/tests/helpers/registrar.rs

@@ -1,77 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use std::str;
-use std::sync::Arc;
-use std::collections::HashMap;
-
-use ethereum_types::{H256, Address};
-use bytes::{Bytes, ToPretty};
-use registrar::{RegistrarClient, Asynchronous};
-use parking_lot::Mutex;
-use rustc_hex::FromHex;
-
-const REGISTRAR: &'static str = "8e4e9b13d4b45cb0befc93c3061b1408f67316b2";
-const URLHINT: &'static str = "deadbeefcafe0000000000000000000000000000";
-const URLHINT_RESOLVE: &'static str = "267b6922";
-const DEFAULT_HASH: &'static str = "1472a9e190620cdf6b31f383373e45efcfe869a820c91f9ccd7eb9fb45e4985d";
-
-pub struct FakeRegistrar {
-	pub calls: Arc<Mutex<Vec<(String, String)>>>,
-	pub responses: Mutex<HashMap<(String, String), Result<Bytes, String>>>,
-}
-
-impl FakeRegistrar {
-	pub fn new() -> Self {
-		FakeRegistrar {
-			calls: Arc::new(Mutex::new(Vec::new())),
-			responses: Mutex::new({
-				let mut map = HashMap::new();
-				map.insert(
-					(REGISTRAR.into(), "6795dbcd058740ee9a5a3fb9f1cfa10752baec87e09cc45cd7027fd54708271aca300c75000000000000000000000000000000000000000000000000000000000000004000000000000000000000000000000000000000000000000000000000000000014100000000000000000000000000000000000000000000000000000000000000".into()),
-					Ok(format!("000000000000000000000000{}", URLHINT).from_hex().unwrap()),
-				);
-				map.insert(
-					(URLHINT.into(), format!("{}{}", URLHINT_RESOLVE, DEFAULT_HASH)),
-					Ok(vec![])
-				);
-				map
-			}),
-		}
-	}
-
-	pub fn set_result(&self, hash: H256, result: Result<Bytes, String>) {
-		self.responses.lock().insert(
-			(URLHINT.into(), format!("{}{:x}", URLHINT_RESOLVE, hash)),
-			result
-		);
-	}
-}
-
-impl RegistrarClient for FakeRegistrar {
-	type Call = Asynchronous;
-
-	fn registrar_address(&self) -> Result<Address, String> {
-		Ok(REGISTRAR.parse().unwrap())
-	}
-
-	fn call_contract(&self, address: Address, data: Bytes) -> Self::Call {
-		let call = (address.to_hex(), data.to_hex());
-		self.calls.lock().push(call.clone());
-		let res = self.responses.lock().get(&call).cloned().expect(&format!("No response for call: {:?}", call));
-		Box::new(::futures::future::done(res))
-	}
-}

dapps/src/tests/mod.rs

@@ -1,24 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Dapps server test suite
-
-mod helpers;
-
-mod api;
-mod fetch;
-mod rpc;
-mod validation;

dapps/src/tests/rpc.rs

@@ -1,49 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use jsonrpc_core::{IoHandler, Value};
-
-use tests::helpers::{serve_with_rpc, request};
-
-#[test]
-fn should_serve_rpc() {
-	// given
-	let mut io = IoHandler::default();
-	io.add_method("rpc_test", |_| {
-		Ok(Value::String("Hello World!".into()))
-	});
-	let server = serve_with_rpc(io);
-
-	// when
-	let req = r#"{"jsonrpc":"2.0","id":1,"method":"rpc_test","params":[]}"#;
-	let response = request(server, &format!(
-		"\
-			POST /rpc/ HTTP/1.1\r\n\
-			Host: 127.0.0.1:8080\r\n\
-			Connection: close\r\n\
-			Content-Type: application/json\r\n\
-			Content-Length: {}\r\n\
-			\r\n\
-			{}\r\n\
-		",
-		req.as_bytes().len(),
-		req,
-	));
-
-	// then
-	response.assert_status("HTTP/1.1 200 OK");
-	assert_eq!(response.body, "31\n{\"jsonrpc\":\"2.0\",\"result\":\"Hello World!\",\"id\":1}\n\n0\n\n".to_owned());
-}

dapps/src/tests/validation.rs

@@ -1,58 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use tests::helpers::{serve_hosts, request};
-
-#[test]
-fn should_reject_invalid_host() {
-	// given
-	let server = serve_hosts(Some(vec!["localhost:8080".into()]));
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: 127.0.0.1:8080\r\n\
-			Connection: close\r\n\
-			\r\n\
-			{}
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 403 Forbidden");
-	assert!(response.body.contains("Provided Host header is not whitelisted."), response.body);
-}
-
-#[test]
-fn should_serve_dapps_domains() {
-	// given
-	let server = serve_hosts(Some(vec!["localhost:8080".into()]));
-
-	// when
-	let response = request(server,
-		"\
-			GET / HTTP/1.1\r\n\
-			Host: proxy.web3.site\r\n\
-			Connection: close\r\n\
-			\r\n\
-			{}
-		"
-	);
-
-	// then
-	response.assert_status("HTTP/1.1 200 OK");
-}

dapps/src/web.rs

@@ -1,161 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Serving web-based content (proxying)
-
-use std::sync::Arc;
-
-use base32;
-use fetch::{self, Fetch};
-use hyper::{mime, StatusCode};
-
-use apps;
-use endpoint::{Endpoint, EndpointPath, Request, Response};
-use futures::future;
-use futures_cpupool::CpuPool;
-use handlers::{
-	ContentFetcherHandler, ContentHandler, ContentValidator, ValidatorResponse,
-	StreamingHandler,
-};
-use WebProxyTokens;
-
-pub struct Web<F> {
-	web_proxy_tokens: Arc<WebProxyTokens>,
-	fetch: F,
-	pool: CpuPool,
-}
-
-impl<F: Fetch> Web<F> {
-	pub fn boxed(
-		web_proxy_tokens: Arc<WebProxyTokens>,
-		fetch: F,
-		pool: CpuPool,
-	) -> Box<Endpoint> {
-		Box::new(Web {
-			web_proxy_tokens,
-			fetch,
-			pool,
-		})
-	}
-
-	fn extract_target_url(&self, path: &EndpointPath) -> Result<String, ContentHandler> {
-		let token_and_url = path.app_params.get(0)
-			.map(|encoded| encoded.replace('.', ""))
-			.and_then(|encoded| base32::decode(base32::Alphabet::Crockford, &encoded.to_uppercase()))
-			.and_then(|data| String::from_utf8(data).ok())
-			.ok_or_else(|| ContentHandler::error(
-				StatusCode::BadRequest,
-				"Invalid parameter",
-				"Couldn't parse given parameter:",
-				path.app_params.get(0).map(String::as_str),
-			))?;
-
-		let mut token_it = token_and_url.split('+');
-		let token = token_it.next();
-		let target_url = token_it.next();
-
-		// Check if token supplied in URL is correct.
-		let domain = match token.and_then(|token| self.web_proxy_tokens.domain(token)) {
-			Some(domain) => domain,
-			_ => {
-				return Err(ContentHandler::error(
-					StatusCode::BadRequest, "Invalid Access Token", "Invalid or old web proxy access token supplied.", Some("Try refreshing the page."),
-				));
-			}
-		};
-
-		// Validate protocol
-		let mut target_url = match target_url {
-			Some(url) if url.starts_with("http://") || url.starts_with("https://") => url.to_owned(),
-			_ => {
-				return Err(ContentHandler::error(
-					StatusCode::BadRequest, "Invalid Protocol", "Invalid protocol used.", None,
-				));
-			}
-		};
-
-		if !target_url.starts_with(&*domain) {
-			return Err(ContentHandler::error(
-				StatusCode::BadRequest, "Invalid Domain", "Dapp attempted to access invalid domain.", Some(&target_url),
-			));
-		}
-
-		if !target_url.ends_with("/") {
-			target_url = format!("{}/", target_url);
-		}
-
-		// Skip the token
-		let query = path.query.as_ref().map_or_else(String::new, |query| format!("?{}", query));
-		let path = path.app_params[1..].join("/");
-
-		Ok(format!("{}{}{}", target_url, path, query))
-	}
-}
-
-impl<F: Fetch> Endpoint for Web<F> {
-	fn respond(&self, path: EndpointPath, req: Request) -> Response {
-		// First extract the URL (reject invalid URLs)
-		let target_url = match self.extract_target_url(&path) {
-			Ok(url) => url,
-			Err(response) => {
-				return Box::new(future::ok(response.into()));
-			}
-		};
-
-		let token = path.app_params.get(0)
-			.expect("`target_url` is valid; app_params is not empty;qed")
-			.to_owned();
-
-		Box::new(ContentFetcherHandler::new(
-			req.method(),
-			&target_url,
-			path,
-			WebInstaller {
-				token,
-			},
-			self.fetch.clone(),
-			self.pool.clone(),
-		))
-	}
-}
-
-struct WebInstaller {
-	token: String,
-}
-
-impl ContentValidator for WebInstaller {
-	type Error = String;
-
-	fn validate_and_install(self, response: fetch::Response) -> Result<ValidatorResponse, String> {
-		let status = response.status();
-		let is_html = response.is_html();
-		let mime = response.content_type().unwrap_or(mime::TEXT_HTML);
-		let mut handler = StreamingHandler::new(
-			fetch::BodyReader::new(response),
-			status,
-			mime,
-		);
-		if is_html {
-			handler.set_initial_content(&format!(
-				r#"<script>history.replaceState({{}}, "", "/?{}{}/{}")</script>"#,
-				apps::URL_REFERER,
-				apps::WEB_PATH,
-				&self.token,
-			));
-		}
-		Ok(ValidatorResponse::Streaming(handler))
-	}
-}

docker/alpine/Dockerfile

@@ -1,11 +1,11 @@
-FROM alpine:3.7
+FROM alpine:edge
 
 WORKDIR /build
 
 # install tools and dependencies
-RUN apk add --no-cache gcc musl-dev openssl-dev pkgconfig g++ make curl \
-					   eudev-dev rust cargo git file binutils libusb-dev \
-					   linux-headers
+RUN apk add --no-cache gcc musl-dev pkgconfig g++ make curl \
+						eudev-dev rust cargo git file binutils \
+						libusb-dev linux-headers perl cmake
 
 # show backtraces
 ENV RUST_BACKTRACE 1

docker/android/Dockerfile

@@ -23,24 +23,6 @@ RUN /usr/local/android-ndk-r16b/build/tools/make-standalone-toolchain.sh \
     --arch=arm --install-dir=/opt/ndk-standalone --stl=libc++ --platform=android-26
 ENV PATH $PATH:/opt/ndk-standalone/bin
 
-# Compiling OpenSSL for Android
-RUN cd /root && \
-    git clone git://git.openssl.org/openssl.git && \
-    cd openssl && \
-    git checkout OpenSSL_1_1_0-stable
-ENV CROSS_SYSROOT /opt/ndk-standalone/sysroot
-RUN cd /root/openssl && \
-    ./Configure android-armeabi --cross-compile-prefix=arm-linux-androideabi- \
-    -static no-stdio no-ui \
-    -I/usr/local/android-ndk-r16b/sysroot/usr/include \
-    -I/usr/local/android-ndk-r16b/sysroot/usr/include/arm-linux-androideabi \
-    -L/usr/local/android-ndk-r16b/sysroot/usr/lib \
-    --prefix=/opt/ndk-standalone/sysroot/usr
-RUN cd /root/openssl && \
-    make build_libs && \
-    make install_dev
-RUN rm -rf /root/openssl
-
 # Compiling libudev for Android
 # This is the most hacky part of the process, as we need to apply a patch and pass specific
 # options that the compiler environment doesn't define.

docker/centos/Dockerfile

@@ -3,7 +3,7 @@ WORKDIR /build
 
 # install tools and dependencies
 RUN yum -y update&& \
-    yum install -y git make gcc-c++ gcc file binutils
+    yum install -y git make gcc-c++ gcc file binutils cmake
 
 # install rustup
 RUN curl -sSf https://static.rust-lang.org/rustup.sh -o rustup.sh &&\

docker/hub/Dockerfile

@@ -1,4 +1,4 @@
-FROM ubuntu:14.04
+FROM ubuntu:xenial
 MAINTAINER Parity Technologies <devops@parity.io>
 WORKDIR /build
 #ENV for build TAG
@@ -13,6 +13,7 @@ RUN apt-get update && \
         # add-apt-repository
         software-properties-common \
         make \
+        cmake \
         curl \
         wget \
         git \
@@ -22,8 +23,6 @@ RUN apt-get update && \
         libc6-dev \
         binutils \
         file \
-        openssl \
-        libssl-dev \
         libudev-dev \
         pkg-config \
         dpkg-dev \
@@ -35,14 +34,13 @@ RUN apt-get update && \
 # show backtraces
  RUST_BACKTRACE=1 && \
 # build parity
-cd /build&&git clone https://github.com/paritytech/parity && \
-        cd parity && \
+cd /build&&git clone https://github.com/paritytech/parity-ethereum && \
+        cd parity-ethereum && \
 	git pull&& \
 	git checkout $BUILD_TAG && \
         cargo build --verbose --release --features final && \
-        #ls /build/parity/target/release/parity && \
-        strip /build/parity/target/release/parity && \
- file /build/parity/target/release/parity&&mkdir -p /parity&& cp /build/parity/target/release/parity /parity&&\
+        strip /build/parity-ethereum/target/release/parity && \
+ file /build/parity-ethereum/target/release/parity&&mkdir -p /parity&& cp /build/parity-ethereum/target/release/parity /parity&&\
 #cleanup Docker image
  rm -rf /root/.cargo&&rm -rf /root/.multirust&&rm -rf /root/.rustup&&rm -rf /build&&\
  apt-get purge -y  \
@@ -51,6 +49,7 @@ cd /build&&git clone https://github.com/paritytech/parity && \
         # add-apt-repository
         software-properties-common \
         make \
+        cmake \
         curl \
         wget \
         git \

docker/ubuntu-aarch64/Dockerfile

@@ -6,7 +6,7 @@ RUN apt-get -y update && \
         apt-get install -y --force-yes --no-install-recommends \
         curl git make g++ gcc-aarch64-linux-gnu g++-aarch64-linux-gnu \
         libc6-arm64-cross libc6-dev-arm64-cross wget file ca-certificates \
-        binutils-aarch64-linux-gnu \
+        binutils-aarch64-linux-gnu cmake \
         && \
     apt-get clean
 

docker/ubuntu-arm/Dockerfile

@@ -6,7 +6,7 @@ RUN apt-get -y update && \
         apt-get install -y --force-yes --no-install-recommends \
         curl git make g++ gcc-arm-linux-gnueabihf g++-arm-linux-gnueabihf \
         libc6-dev-armhf-cross wget file ca-certificates \
-        binutils-arm-linux-gnueabihf \
+        binutils-arm-linux-gnueabihf cmake \
         && \
     apt-get clean
 

docker/ubuntu/Dockerfile

@@ -6,11 +6,11 @@ RUN apt-get update && \
         apt-get install -y \
         g++ \
         build-essential \
+        cmake \
         curl \
         git \
         file \
         binutils \
-        libssl-dev \
         pkg-config \
         libudev-dev
 

docs/CHANGELOG-1.10.md

@@ -1,3 +1,84 @@
+## Parity [v1.10.9](https://github.com/paritytech/parity/releases/tag/v1.10.9) (2018-07-07)
+
+Parity 1.10.9 is a bug-fix release to improve performance and stability.
+
+The full list of included changes:
+
+- Stable: 1.10.9 backports ([#9016](https://github.com/paritytech/parity/pull/9016))
+  - Parity-version: bump stable to 1.10.9
+  - Scripts: remove md5 checksums ([#8884](https://github.com/paritytech/parity/pull/8884))
+  - Add support for --chain tobalaba ([#8870](https://github.com/paritytech/parity/pull/8870))
+    - Add support for --chain tobalaba
+  - Only return error log for rustls ([#9025](https://github.com/paritytech/parity/pull/9025))
+  - Fixes for misbehavior reporting in AuthorityRound ([#8998](https://github.com/paritytech/parity/pull/8998))
+    - Aura: only report after checking for repeated skipped primaries
+    - Aura: refactor duplicate code for getting epoch validator set
+    - Aura: verify_external: report on validator set contract instance
+    - Aura: use correct validator set epoch number when reporting
+    - Aura: use epoch set when verifying blocks
+    - Aura: report skipped primaries when generating seal
+    - Aura: handle immediate transitions
+    - Aura: don't report skipped steps from genesis to first block
+    - Aura: fix reporting test
+    - Aura: refactor duplicate code to handle immediate_transitions
+    - Aura: let reporting fail on verify_block_basic
+    - Aura: add comment about possible failure of reporting
+
+## Parity [v1.10.8](https://github.com/paritytech/parity/releases/tag/v1.10.8) (2018-06-29)
+
+Parity 1.10.8 is a bug-fix release to improve performance and stability.
+
+The full list of included changes:
+
+- Backports ([#8986](https://github.com/paritytech/parity/pull/8986))
+  - Snap: downgrade rust to revision 1.26.2, ref snapcraft/+bug/1778530 ([#8984](https://github.com/paritytech/parity/pull/8984))
+    - Snap: downgrade rust to revision 1.26.2, ref snapcraft/+bug/1778530
+    - Snap: use plugin rust
+  - Fix deadlock in blockchain. ([#8977](https://github.com/paritytech/parity/pull/8977))
+  - Remove js-glue from workspace
+- Bump stable to 1.10.8 ([#8951](https://github.com/paritytech/parity/pull/8951))
+  - Parity-version: bump stable to 1.10.8
+  - Update ropsten.json ([#8926](https://github.com/paritytech/parity/pull/8926))
+  - Scripts: minor improvements ([#8930](https://github.com/paritytech/parity/pull/8930))
+    - CI: enable 'latest' docker tag on master pipeline
+    - CI: mark both beta and stable as stable snap.
+    - CI: sign all windows binaries
+    - Scripts: remove whisper target not available in stable
+  - Scripts: fix gitlab strip binaries
+  - Scripts: fix docker build tag on latest using master ([#8952](https://github.com/paritytech/parity/pull/8952))
+  - Rpc: cap gas limit of local calls ([#8943](https://github.com/paritytech/parity/pull/8943))
+
+## Parity [v1.10.7](https://github.com/paritytech/parity/releases/tag/v1.10.7) (2018-06-20)
+
+Parity 1.10.7 is a bug-fix release to improve performance and stability.
+
+The full list of included changes:
+
+- Backports ([#8919](https://github.com/paritytech/parity/pull/8919))
+  - Fixed AuthorityRound deadlock on shutdown, closes [#8088](https://github.com/paritytech/parity/issues/8088) ([#8803](https://github.com/paritytech/parity/pull/8803))
+  - CI: Fix docker tags ([#8822](https://github.com/paritytech/parity/pull/8822))
+    - Scripts: enable docker builds for beta and stable
+    - Scripts: docker latest should be beta not master
+    - Scripts: docker latest is master
+  - Fix concurrent access to signer queue ([#8854](https://github.com/paritytech/parity/pull/8854))
+    - Fix concurrent access to signer queue
+    - Put request back to the queue if confirmation failed
+    - Typo: fix docs and rename functions to be more specific
+    - Change trace info "Transaction" -> "Request"
+  - Add new ovh bootnodes and fix port for foundation bootnode 3.2 ([#8886](https://github.com/paritytech/parity/pull/8886))
+    - Add new ovh bootnodes and fix port for foundation bootnode 3.2
+    - Remove old bootnodes.
+    - Remove duplicate 1118980bf48b0a3640bdba04e0fe78b1add18e1cd99bf22d53daac1fd9972ad650df52176e7c7d89d1114cfef2bc23a2959aa54998a46afcf7d91809f0855082
+  - Block 0 is valid in queries ([#8891](https://github.com/paritytech/parity/pull/8891))
+  - Update jsonrpc libs, fixed ipc leak, closes [#8774](https://github.com/paritytech/parity/issues/8774) ([#8876](https://github.com/paritytech/parity/pull/8876))
+  - Add ETC Cooperative-run load balanced parity node ([#8892](https://github.com/paritytech/parity/pull/8892))
+  - Minor fix in chain supplier and light provider ([#8906](https://github.com/paritytech/parity/pull/8906))
+    - Fix chain supplier increment
+    - Fix light provider block_headers
+- Parity-version: stable release 1.10.7 ([#8855](https://github.com/paritytech/parity/pull/8855))
+  - Cherry-pick network-specific release flag ([#8821](https://github.com/paritytech/parity/pull/8821))
+  - Parity-version: bump stable to 1.10.7
+
 ## Parity [v1.10.6](https://github.com/paritytech/parity/releases/tag/v1.10.6) (2018-06-05)
 
 Parity 1.10.6 is a security-relevant release. Please upgrade your nodes as soon as possible.

ethash/Cargo.toml

@@ -6,13 +6,14 @@ authors = ["Parity Technologies <admin@parity.io>"]
 [lib]
 
 [dependencies]
-log = "0.3"
-keccak-hash = { path = "../util/hash" }
-primal = "0.2.3"
-parking_lot = "0.5"
 crunchy = "0.1.0"
-memmap = "0.6"
 either = "1.0.0"
+ethereum-types = "0.3"
+keccak-hash = { git = "https://github.com/paritytech/parity-common" }
+log = "0.4"
+memmap = "0.6"
+parking_lot = "0.6"
+primal = "0.2.3"
 
 [dev-dependencies]
 tempdir = "0.3"

ethash/src/cache.rs

@@ -91,7 +91,7 @@ impl NodeCacheBuilder {
 
 	pub fn new<T: Into<Option<OptimizeFor>>>(optimize_for: T) -> Self {
 		NodeCacheBuilder {
-			seedhash: Arc::new(Mutex::new(SeedHashCompute::new())),
+			seedhash: Arc::new(Mutex::new(SeedHashCompute::default())),
 			optimize_for: optimize_for.into().unwrap_or_default(),
 		}
 	}

ethash/src/compute.rs

@@ -137,11 +137,11 @@ fn hash_compute(light: &Light, full_size: usize, header_hash: &H256, nonce: u64)
 		($n:expr, $value:expr) => {{
 			// We use explicit lifetimes to ensure that val's borrow is invalidated until the
 			// transmuted val dies.
-			unsafe fn make_const_array<'a, T, U>(val: &'a mut [T]) -> &'a mut [U; $n] {
+			unsafe fn make_const_array<T, U>(val: &mut [T]) -> &mut [U; $n] {
 				use ::std::mem;
 
 				debug_assert_eq!(val.len() * mem::size_of::<T>(), $n * mem::size_of::<U>());
-				mem::transmute(val.as_mut_ptr())
+ 				&mut *(val.as_mut_ptr() as *mut [U; $n])
 			}
 
 			make_const_array($value)
@@ -177,7 +177,7 @@ fn hash_compute(light: &Light, full_size: usize, header_hash: &H256, nonce: u64)
 
 			ptr::copy_nonoverlapping(header_hash.as_ptr(), out.as_mut_ptr(), header_hash.len());
 			ptr::copy_nonoverlapping(
-				mem::transmute(&nonce),
+				&nonce as *const u64 as *const u8,
 				out[header_hash.len()..].as_mut_ptr(),
 				mem::size_of::<u64>(),
 			);
@@ -266,18 +266,20 @@ fn hash_compute(light: &Light, full_size: usize, header_hash: &H256, nonce: u64)
 
 	let mix_hash = buf.compress_bytes;
 
-	let value: H256 = unsafe {
+	let value: H256 = {
 		// We can interpret the buffer as an array of `u8`s, since it's `repr(C)`.
-		let read_ptr: *const u8 = mem::transmute(&buf);
+		let read_ptr: *const u8 = &buf as *const MixBuf as *const u8;
 		// We overwrite the second half since `keccak_256` has an internal buffer and so allows
 		// overlapping arrays as input.
-		let write_ptr: *mut u8 = mem::transmute(&mut buf.compress_bytes);
-		keccak_256::unchecked(
-			write_ptr,
-			buf.compress_bytes.len(),
-			read_ptr,
-			buf.half_mix.bytes.len() + buf.compress_bytes.len(),
-		);
+		let write_ptr: *mut u8 = &mut buf.compress_bytes as *mut [u8; 32] as *mut u8;
+		unsafe { 
+			keccak_256::unchecked(
+				write_ptr,
+				buf.compress_bytes.len(),
+				read_ptr,
+				buf.half_mix.bytes.len() + buf.compress_bytes.len(),
+			);
+		}
 		buf.compress_bytes
 	};
 

ethash/src/lib.rs

@@ -16,10 +16,11 @@
 
 #![cfg_attr(feature = "benches", feature(test))]
 
-extern crate primal;
-extern crate parking_lot;
 extern crate either;
+extern crate ethereum_types;
 extern crate memmap;
+extern crate parking_lot;
+extern crate primal;
 
 #[macro_use]
 extern crate crunchy;
@@ -38,6 +39,7 @@ mod shared;
 pub use cache::{NodeCacheBuilder, OptimizeFor};
 pub use compute::{ProofOfWork, quick_get_difficulty, slow_hash_block_number};
 use compute::Light;
+use ethereum_types::{U256, U512};
 use keccak::H256;
 use parking_lot::Mutex;
 pub use seed_compute::SeedHashCompute;
@@ -136,6 +138,29 @@ impl EthashManager {
 	}
 }
 
+/// Convert an Ethash boundary to its original difficulty. Basically just `f(x) = 2^256 / x`.
+pub fn boundary_to_difficulty(boundary: &ethereum_types::H256) -> U256 {
+	difficulty_to_boundary_aux(&**boundary)
+}
+
+/// Convert an Ethash difficulty to the target boundary. Basically just `f(x) = 2^256 / x`.
+pub fn difficulty_to_boundary(difficulty: &U256) -> ethereum_types::H256 {
+	difficulty_to_boundary_aux(difficulty).into()
+}
+
+fn difficulty_to_boundary_aux<T: Into<U512>>(difficulty: T) -> ethereum_types::U256 {
+	let difficulty = difficulty.into();
+
+	assert!(!difficulty.is_zero());
+
+	if difficulty == U512::one() {
+		U256::max_value()
+	} else {
+		// difficulty > 1, so result should never overflow 256 bits
+		U256::from((U512::one() << 256) / difficulty)
+	}
+}
+
 #[test]
 fn test_lru() {
 	use tempdir::TempDir;
@@ -155,6 +180,43 @@ fn test_lru() {
 	assert_eq!(ethash.cache.lock().prev_epoch.unwrap(), 0);
 }
 
+#[test]
+fn test_difficulty_to_boundary() {
+	use ethereum_types::H256;
+	use std::str::FromStr;
+
+	assert_eq!(difficulty_to_boundary(&U256::from(1)), H256::from(U256::max_value()));
+	assert_eq!(difficulty_to_boundary(&U256::from(2)), H256::from_str("8000000000000000000000000000000000000000000000000000000000000000").unwrap());
+	assert_eq!(difficulty_to_boundary(&U256::from(4)), H256::from_str("4000000000000000000000000000000000000000000000000000000000000000").unwrap());
+	assert_eq!(difficulty_to_boundary(&U256::from(32)), H256::from_str("0800000000000000000000000000000000000000000000000000000000000000").unwrap());
+}
+
+#[test]
+fn test_difficulty_to_boundary_regression() {
+	use ethereum_types::H256;
+
+	// the last bit was originally being truncated when performing the conversion
+	// https://github.com/paritytech/parity-ethereum/issues/8397
+	for difficulty in 1..9 {
+		assert_eq!(U256::from(difficulty), boundary_to_difficulty(&difficulty_to_boundary(&difficulty.into())));
+		assert_eq!(H256::from(difficulty), difficulty_to_boundary(&boundary_to_difficulty(&difficulty.into())));
+		assert_eq!(U256::from(difficulty), boundary_to_difficulty(&boundary_to_difficulty(&difficulty.into()).into()));
+		assert_eq!(H256::from(difficulty), difficulty_to_boundary(&difficulty_to_boundary(&difficulty.into()).into()));
+	}
+}
+
+#[test]
+#[should_panic]
+fn test_difficulty_to_boundary_panics_on_zero() {
+	difficulty_to_boundary(&U256::from(0));
+}
+
+#[test]
+#[should_panic]
+fn test_boundary_to_difficulty_panics_on_zero() {
+	boundary_to_difficulty(&ethereum_types::H256::from(0));
+}
+
 #[cfg(feature = "benches")]
 mod benchmarks {
 	extern crate test;

ethash/src/seed_compute.rs

@@ -19,20 +19,13 @@ use keccak::{keccak_256, H256};
 
 use std::cell::Cell;
 
+#[derive(Default)]
 pub struct SeedHashCompute {
 	prev_epoch: Cell<u64>,
 	prev_seedhash: Cell<H256>,
 }
 
 impl SeedHashCompute {
-	#[inline]
-	pub fn new() -> SeedHashCompute {
-		SeedHashCompute {
-			prev_epoch: Cell::new(0),
-			prev_seedhash: Cell::new([0u8; 32]),
-		}
-	}
-
 	#[inline]
 	fn reset_cache(&self) {
 		self.prev_epoch.set(0);
@@ -77,20 +70,20 @@ mod tests {
 
 	#[test]
 	fn test_seed_compute_once() {
-		let seed_compute = SeedHashCompute::new();
+		let seed_compute = SeedHashCompute::default();
 		let hash = [241, 175, 44, 134, 39, 121, 245, 239, 228, 236, 43, 160, 195, 152, 46, 7, 199, 5, 253, 147, 241, 206, 98, 43, 3, 104, 17, 40, 192, 79, 106, 162];
 		assert_eq!(seed_compute.hash_block_number(486382), hash);
 	}
 
 	#[test]
 	fn test_seed_compute_zero() {
-		let seed_compute = SeedHashCompute::new();
+		let seed_compute = SeedHashCompute::default();
 		assert_eq!(seed_compute.hash_block_number(0), [0u8; 32]);
 	}
 
 	#[test]
 	fn test_seed_compute_after_older() {
-		let seed_compute = SeedHashCompute::new();
+		let seed_compute = SeedHashCompute::default();
 		// calculating an older value first shouldn't affect the result
 		let _ = seed_compute.hash_block_number(50000);
 		let hash = [241, 175, 44, 134, 39, 121, 245, 239, 228, 236, 43, 160, 195, 152, 46, 7, 199, 5, 253, 147, 241, 206, 98, 43, 3, 104, 17, 40, 192, 79, 106, 162];
@@ -99,7 +92,7 @@ mod tests {
 
 	#[test]
 	fn test_seed_compute_after_newer() {
-		let seed_compute = SeedHashCompute::new();
+		let seed_compute = SeedHashCompute::default();
 		// calculating an newer value first shouldn't affect the result
 		let _ = seed_compute.hash_block_number(972764);
 		let hash = [241, 175, 44, 134, 39, 121, 245, 239, 228, 236, 43, 160, 195, 152, 46, 7, 199, 5, 253, 147, 241, 206, 98, 43, 3, 104, 17, 40, 192, 79, 106, 162];

ethcore/Cargo.toml

@@ -8,24 +8,24 @@ authors = ["Parity Technologies <admin@parity.io>"]
 
 [dependencies]
 ansi_term = "0.10"
-bloomchain = { path = "../util/bloomchain" }
+blooms-db = { path = "../util/blooms-db" }
 bn = { git = "https://github.com/paritytech/bn", default-features = false }
 byteorder = "1.0"
 common-types = { path = "types" }
 crossbeam = "0.3"
 ethash = { path = "../ethash" }
 ethcore-bloom-journal = { path = "../util/bloom" }
-ethcore-bytes = { path = "../util/bytes" }
-fetch = { path = "../util/fetch" }
-hashdb = { path = "../util/hashdb" }
-memorydb = { path = "../util/memorydb" }
-patricia-trie = { path = "../util/patricia_trie" }
-ethcore-crypto = { path = "crypto" }
-error-chain = { version = "0.11", default-features = false }
+parity-bytes = { git = "https://github.com/paritytech/parity-common" }
+hashdb = { git = "https://github.com/paritytech/parity-common" }
+memorydb = { git = "https://github.com/paritytech/parity-common" }
+patricia-trie = { git = "https://github.com/paritytech/parity-common" }
+patricia-trie-ethereum = { path = "../util/patricia-trie-ethereum" }
+parity-crypto = { git = "https://github.com/paritytech/parity-common" }
+error-chain = { version = "0.12", default-features = false }
 ethcore-io = { path = "../util/io" }
 ethcore-logger = { path = "../logger" }
 ethcore-miner = { path = "../miner" }
-ethcore-stratum = { path = "./stratum" }
+ethcore-stratum = { path = "./stratum", optional = true }
 ethcore-transaction = { path = "./transaction" }
 ethereum-types = "0.3"
 memory-cache = { path = "../util/memory_cache" }
@@ -36,7 +36,6 @@ ethjson = { path = "../json" }
 ethkey = { path = "../ethkey" }
 ethstore = { path = "../ethstore" }
 evm = { path = "evm" }
-hardware-wallet = { path = "../hw" }
 heapsize = "0.4"
 itertools = "0.5"
 lazy_static = "1.0"
@@ -45,15 +44,14 @@ lru-cache = "0.1"
 num = { version = "0.1", default-features = false, features = ["bigint"] }
 num_cpus = "1.2"
 parity-machine = { path = "../machine" }
-parking_lot = "0.5"
+parking_lot = "0.6"
 rayon = "1.0"
 rand = "0.4"
-rlp = { path = "../util/rlp" }
+rlp = { git = "https://github.com/paritytech/parity-common" }
 rlp_compress = { path = "../util/rlp_compress" }
 rlp_derive = { path = "../util/rlp_derive" }
-kvdb = { path = "../util/kvdb" }
-kvdb-memorydb = { path = "../util/kvdb-memorydb" }
-util-error = { path = "../util/error" }
+kvdb = { git = "https://github.com/paritytech/parity-common" }
+kvdb-memorydb = { git = "https://github.com/paritytech/parity-common" }
 snappy = { git = "https://github.com/paritytech/rust-snappy" }
 stop-guard = { path = "../util/stop-guard" }
 macros = { path = "../util/macros" }
@@ -63,19 +61,38 @@ trace-time = { path = "../util/trace-time" }
 using_queue = { path = "../util/using_queue" }
 vm = { path = "vm" }
 wasm = { path = "wasm" }
-keccak-hash = { path = "../util/hash" }
-triehash = { path = "../util/triehash" }
+keccak-hash = { git = "https://github.com/paritytech/parity-common" }
+triehash-ethereum = { version = "0.2",  path = "../util/triehash-ethereum" }
 unexpected = { path = "../util/unexpected" }
 journaldb = { path = "../util/journaldb" }
+keccak-hasher = { path = "../util/keccak-hasher" }
+kvdb-rocksdb = { git = "https://github.com/paritytech/parity-common" }
+tempdir = {version="0.3", optional = true}
+
+[target.'cfg(any(target_os = "linux", target_os = "macos", target_os = "windows", target_os = "android"))'.dependencies]
+hardware-wallet = { path = "../hw" }
+
+[target.'cfg(not(any(target_os = "linux", target_os = "macos", target_os = "windows", target_os = "android")))'.dependencies]
+fake-hardware-wallet = { path = "../util/fake-hardware-wallet" }
 
 [dev-dependencies]
 tempdir = "0.3"
-trie-standardmap = { path = "../util/trie-standardmap" }
-kvdb-rocksdb = { path = "../util/kvdb-rocksdb" }
+trie-standardmap = { git = "https://github.com/paritytech/parity-common" }
 
 [features]
+parity = ["work-notify", "price-info", "stratum"]
+# Large optional features that are enabled by default for Parity,
+# but might be omitted for other dependent crates.
+work-notify = ["ethcore-miner/work-notify"]
+price-info = ["ethcore-miner/price-info"]
+stratum = ["ethcore-stratum"]
+
+# Disables seal verification for mined blocks.
+# This allows you to submit any seal via RPC to test and benchmark
+# how fast pending block get's created while running on the mainnet.
+miner-debug = []
 # Display EVM debug traces.
-evm-debug = ["slow-blocks"]
+evm-debug = ["evm/evm-debug"]
 # Display EVM debug traces when running tests.
 evm-debug-tests = ["evm-debug", "evm/evm-debug-tests"]
 # Measure time of transaction execution.
@@ -84,10 +101,10 @@ evm-debug-tests = ["evm-debug", "evm/evm-debug-tests"]
 # EVM debug traces are printed.
 slow-blocks = []
 # Run JSON consensus tests.
-json-tests = ["ethcore-transaction/json-tests"]
+json-tests = ["ethcore-transaction/json-tests", "test-helpers", "tempdir"]
 # Run memory/cpu heavy tests.
 test-heavy = []
 # Compile benches
 benches = []
 # Compile test helpers
-test-helpers = []
+test-helpers = ["tempdir"]

ethcore/benches/evm.rs

@@ -16,17 +16,16 @@
 
 #![feature(test)]
 
-extern crate test;
-extern crate ethcore_util as util;
-extern crate rand;
 extern crate bn;
-extern crate ethcore_crypto;
+extern crate ethereum_types;
 extern crate ethkey;
+extern crate parity_crypto;
+extern crate rand;
 extern crate rustc_hex;
-extern crate ethcore_bigint;
+extern crate test;
 
-use self::test::{Bencher};
-use rand::{StdRng};
+use self::test::Bencher;
+use rand::StdRng;
 
 #[bench]
 fn bn_128_pairing(b: &mut Bencher) {
@@ -60,10 +59,9 @@ fn bn_128_mul(b: &mut Bencher) {
 
 #[bench]
 fn sha256(b: &mut Bencher) {
-	use ethcore_crypto::digest::sha256;
+	use parity_crypto::digest::sha256;
 
-	let mut input: [u8; 256] = [0; 256];
-	let mut out = [0; 32];
+	let input = [0_u8; 256];
 
 	b.iter(|| {
 		sha256(&input);
@@ -74,7 +72,7 @@ fn sha256(b: &mut Bencher) {
 fn ecrecover(b: &mut Bencher) {
 	use rustc_hex::FromHex;
 	use ethkey::{Signature, recover as ec_recover};
-	use ethcore_bigint::hash::H256;
+	use ethereum_types::H256;
 	let input = FromHex::from_hex("47173285a8d7341e5e972fc677286384f802f8ef42a5ec5f03bbfa254cb01fad000000000000000000000000000000000000000000000000000000000000001b650acf9d3f5f0a2c799776a1254355d5f4061762a237396a99a0e0e3fc2bcd6729514a0dacb2e623ac4abd157cb18163ff942280db4d5caad66ddf941ba12e03").unwrap();
 	let hash = H256::from_slice(&input[0..32]);
 	let v = H256::from_slice(&input[32..64]);

ethcore/crypto/Cargo.toml

@@ -1,12 +0,0 @@
-[package]
-name = "ethcore-crypto"
-version = "0.1.0"
-authors = ["Parity Technologies <admin@parity.io>"]
-
-[dependencies]
-ethereum-types = "0.3"
-quick-error = "1.2"
-ring = "0.12"
-rust-crypto = "0.2.36"
-tiny-keccak = "1.4"
-

ethcore/crypto/README.md

@@ -1,5 +0,0 @@
-# Ethcrypto
-
-General cryptographic utilities for Ethereum.
-
-By default, this library is compiled with the `secp256k1` feature, which provides ECDH and ECIES capability on that curve. It can be compiled without to avoid a dependency on the `libsecp256k1` library.

ethcore/crypto/src/aes.rs

@@ -1,53 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use error::SymmError;
-use rcrypto::blockmodes::{CtrMode, CbcDecryptor, PkcsPadding};
-use rcrypto::aessafe::{AesSafe128Encryptor, AesSafe128Decryptor};
-use rcrypto::symmetriccipher::{Encryptor, Decryptor};
-use rcrypto::buffer::{RefReadBuffer, RefWriteBuffer, WriteBuffer};
-
-/// Encrypt a message (CTR mode).
-///
-/// Key (`k`) length and initialisation vector (`iv`) length have to be 16 bytes each.
-/// An error is returned if the input lengths are invalid.
-pub fn encrypt_128_ctr(k: &[u8], iv: &[u8], plain: &[u8], dest: &mut [u8]) -> Result<(), SymmError> {
-	let mut encryptor = CtrMode::new(AesSafe128Encryptor::new(k), iv.to_vec());
-	encryptor.encrypt(&mut RefReadBuffer::new(plain), &mut RefWriteBuffer::new(dest), true)?;
-	Ok(())
-}
-
-/// Decrypt a message (CTR mode).
-///
-/// Key (`k`) length and initialisation vector (`iv`) length have to be 16 bytes each.
-/// An error is returned if the input lengths are invalid.
-pub fn decrypt_128_ctr(k: &[u8], iv: &[u8], encrypted: &[u8], dest: &mut [u8]) -> Result<(), SymmError> {
-	let mut encryptor = CtrMode::new(AesSafe128Encryptor::new(k), iv.to_vec());
-	encryptor.decrypt(&mut RefReadBuffer::new(encrypted), &mut RefWriteBuffer::new(dest), true)?;
-	Ok(())
-}
-
-/// Decrypt a message (CBC mode).
-///
-/// Key (`k`) length and initialisation vector (`iv`) length have to be 16 bytes each.
-/// An error is returned if the input lengths are invalid.
-pub fn decrypt_128_cbc(k: &[u8], iv: &[u8], encrypted: &[u8], dest: &mut [u8]) -> Result<usize, SymmError> {
-	let mut encryptor = CbcDecryptor::new(AesSafe128Decryptor::new(k), PkcsPadding, iv.to_vec());
-	let len = dest.len();
-	let mut buffer = RefWriteBuffer::new(dest);
-	encryptor.decrypt(&mut RefReadBuffer::new(encrypted), &mut buffer, true)?;
-	Ok(len - buffer.remaining())
-}

ethcore/crypto/src/aes_gcm.rs

@@ -1,198 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use error::SymmError;
-use ring;
-
-enum Mode { Aes128Gcm, Aes256Gcm }
-
-/// AES GCM encryptor.
-pub struct Encryptor<'a> {
-	mode: Mode,
-	key: ring::aead::SealingKey,
-	ad: &'a [u8],
-	offset: usize,
-}
-
-impl<'a> Encryptor<'a> {
-	pub fn aes_128_gcm(key: &[u8; 16]) -> Result<Encryptor<'a>, SymmError> {
-		let sk = ring::aead::SealingKey::new(&ring::aead::AES_128_GCM, key)?;
-		Ok(Encryptor {
-			mode: Mode::Aes128Gcm,
-			key: sk,
-			ad: &[],
-			offset: 0,
-		})
-	}
-
-	pub fn aes_256_gcm(key: &[u8; 32]) -> Result<Encryptor<'a>, SymmError> {
-		let sk = ring::aead::SealingKey::new(&ring::aead::AES_256_GCM, key)?;
-		Ok(Encryptor {
-			mode: Mode::Aes256Gcm,
-			key: sk,
-			ad: &[],
-			offset: 0,
-		})
-	}
-
-	/// Optional associated data which is not encrypted but authenticated.
-	pub fn associate(&mut self, data: &'a [u8]) -> &mut Self {
-		self.ad = data;
-		self
-	}
-
-	/// Optional offset value. Only the slice `[offset..]` will be encrypted.
-	pub fn offset(&mut self, off: usize) -> &mut Self {
-		self.offset = off;
-		self
-	}
-
-	/// Please note that the pair (key, nonce) must never be reused. Using random nonces
-	/// limits the number of messages encrypted with the same key to 2^32 (cf. [[1]])
-	///
-	/// [1]: https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-38d.pdf
-	pub fn encrypt(&self, nonce: &[u8; 12], mut data: Vec<u8>) -> Result<Vec<u8>, SymmError> {
-		if self.offset > data.len() {
-			return Err(SymmError::offset_error(self.offset))
-		}
-		let tag_len = match self.mode {
-			Mode::Aes128Gcm => ring::aead::AES_128_GCM.tag_len(),
-			Mode::Aes256Gcm => ring::aead::AES_256_GCM.tag_len(),
-		};
-		data.extend(::std::iter::repeat(0).take(tag_len));
-		let len = ring::aead::seal_in_place(&self.key, nonce, self.ad, &mut data[self.offset ..], tag_len)?;
-		data.truncate(self.offset + len);
-		Ok(data)
-	}
-}
-
-/// AES GCM decryptor.
-pub struct Decryptor<'a> {
-	key: ring::aead::OpeningKey,
-	ad: &'a [u8],
-	offset: usize,
-}
-
-impl<'a> Decryptor<'a> {
-	pub fn aes_128_gcm(key: &[u8; 16]) -> Result<Decryptor<'a>, SymmError> {
-		let ok = ring::aead::OpeningKey::new(&ring::aead::AES_128_GCM, key)?;
-		Ok(Decryptor {
-			key: ok,
-			ad: &[],
-			offset: 0,
-		})
-	}
-
-	pub fn aes_256_gcm(key: &[u8; 32]) -> Result<Decryptor<'a>, SymmError> {
-		let ok = ring::aead::OpeningKey::new(&ring::aead::AES_256_GCM, key)?;
-		Ok(Decryptor {
-			key: ok,
-			ad: &[],
-			offset: 0,
-		})
-	}
-
-	/// Optional associated data which is not encrypted but authenticated.
-	pub fn associate(&mut self, data: &'a [u8]) -> &mut Self {
-		self.ad = data;
-		self
-	}
-
-	/// Optional offset value. Only the slice `[offset..]` will be decrypted.
-	pub fn offset(&mut self, off: usize) -> &mut Self {
-		self.offset = off;
-		self
-	}
-
-	pub fn decrypt(&self, nonce: &[u8; 12], mut data: Vec<u8>) -> Result<Vec<u8>, SymmError> {
-		if self.offset > data.len() {
-			return Err(SymmError::offset_error(self.offset))
-		}
-		let len = ring::aead::open_in_place(&self.key, nonce, self.ad, 0, &mut data[self.offset ..])?.len();
-		data.truncate(self.offset + len);
-		Ok(data)
-	}
-}
-
-#[cfg(test)]
-mod tests {
-	use super::{Encryptor, Decryptor};
-
-	#[test]
-	fn aes_gcm_128() {
-		let secret = b"1234567890123456";
-		let nonce = b"123456789012";
-		let message = b"So many books, so little time";
-
-		let ciphertext = Encryptor::aes_128_gcm(secret)
-			.unwrap()
-			.encrypt(nonce, message.to_vec())
-			.unwrap();
-
-		assert!(ciphertext != message);
-
-		let plaintext = Decryptor::aes_128_gcm(secret)
-			.unwrap()
-			.decrypt(nonce, ciphertext)
-			.unwrap();
-
-		assert_eq!(plaintext, message)
-	}
-
-	#[test]
-	fn aes_gcm_256() {
-		let secret = b"12345678901234567890123456789012";
-		let nonce = b"123456789012";
-		let message = b"So many books, so little time";
-
-		let ciphertext = Encryptor::aes_256_gcm(secret)
-			.unwrap()
-			.encrypt(nonce, message.to_vec())
-			.unwrap();
-
-		assert!(ciphertext != message);
-
-		let plaintext = Decryptor::aes_256_gcm(secret)
-			.unwrap()
-			.decrypt(nonce, ciphertext)
-			.unwrap();
-
-		assert_eq!(plaintext, message)
-	}
-
-	#[test]
-	fn aes_gcm_256_offset() {
-		let secret = b"12345678901234567890123456789012";
-		let nonce = b"123456789012";
-		let message = b"prefix data; So many books, so little time";
-
-		let ciphertext = Encryptor::aes_256_gcm(secret)
-			.unwrap()
-			.offset(13) // length of "prefix data; "
-			.encrypt(nonce, message.to_vec())
-			.unwrap();
-
-		assert!(ciphertext != &message[..]);
-
-		let plaintext = Decryptor::aes_256_gcm(secret)
-			.unwrap()
-			.offset(13) // length of "prefix data; "
-			.decrypt(nonce, ciphertext)
-			.unwrap();
-
-		assert_eq!(plaintext, &message[..])
-	}
-}

ethcore/crypto/src/digest.rs

@@ -1,109 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use rcrypto::ripemd160;
-use ring::digest::{self, Context, SHA256, SHA512};
-use std::marker::PhantomData;
-use std::ops::Deref;
-
-/// The message digest.
-pub struct Digest<T>(InnerDigest, PhantomData<T>);
-
-enum InnerDigest {
-	Ring(digest::Digest),
-	Ripemd160([u8; 20]),
-}
-
-impl<T> Deref for Digest<T> {
-	type Target = [u8];
-	fn deref(&self) -> &Self::Target {
-		match self.0 {
-			InnerDigest::Ring(ref d) => d.as_ref(),
-			InnerDigest::Ripemd160(ref d) => &d[..]
-		}
-	}
-}
-
-/// Single-step sha256 digest computation.
-pub fn sha256(data: &[u8]) -> Digest<Sha256> {
-	Digest(InnerDigest::Ring(digest::digest(&SHA256, data)), PhantomData)
-}
-
-/// Single-step sha512 digest computation.
-pub fn sha512(data: &[u8]) -> Digest<Sha512> {
-	Digest(InnerDigest::Ring(digest::digest(&SHA512, data)), PhantomData)
-}
-
-/// Single-step ripemd160 digest computation.
-pub fn ripemd160(data: &[u8]) -> Digest<Ripemd160> {
-	let mut hasher = Hasher::ripemd160();
-	hasher.update(data);
-	hasher.finish()
-}
-
-pub enum Sha256 {}
-pub enum Sha512 {}
-pub enum Ripemd160 {}
-
-/// Stateful digest computation.
-pub struct Hasher<T>(Inner, PhantomData<T>);
-
-enum Inner {
-	Ring(Context),
-	Ripemd160(ripemd160::Ripemd160)
-}
-
-impl Hasher<Sha256> {
-	pub fn sha256() -> Hasher<Sha256> {
-		Hasher(Inner::Ring(Context::new(&SHA256)), PhantomData)
-	}
-}
-
-impl Hasher<Sha512> {
-	pub fn sha512() -> Hasher<Sha512> {
-		Hasher(Inner::Ring(Context::new(&SHA512)), PhantomData)
-	}
-}
-
-impl Hasher<Ripemd160> {
-	pub fn ripemd160() -> Hasher<Ripemd160> {
-		Hasher(Inner::Ripemd160(ripemd160::Ripemd160::new()), PhantomData)
-	}
-}
-
-impl<T> Hasher<T> {
-	pub fn update(&mut self, data: &[u8]) {
-		match self.0 {
-			Inner::Ring(ref mut ctx) => ctx.update(data),
-			Inner::Ripemd160(ref mut ctx) => {
-				use rcrypto::digest::Digest;
-				ctx.input(data)
-			}
-		}
-	}
-
-	pub fn finish(self) -> Digest<T> {
-		match self.0 {
-			Inner::Ring(ctx) => Digest(InnerDigest::Ring(ctx.finish()), PhantomData),
-			Inner::Ripemd160(mut ctx) => {
-				use rcrypto::digest::Digest;
-				let mut d = [0; 20];
-				ctx.result(&mut d);
-				Digest(InnerDigest::Ripemd160(d), PhantomData)
-			}
-		}
-	}
-}

ethcore/crypto/src/error.rs

@@ -1,82 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use rcrypto;
-use ring;
-
-quick_error! {
-	#[derive(Debug)]
-	pub enum Error {
-		Scrypt(e: ScryptError) {
-			cause(e)
-			from()
-		}
-		Symm(e: SymmError) {
-			cause(e)
-			from()
-		}
-	}
-}
-
-quick_error! {
-	#[derive(Debug)]
-	pub enum ScryptError {
-		// log(N) < r / 16
-		InvalidN {
-			display("Invalid N argument of the scrypt encryption")
-		}
-		// p <= (2^31-1 * 32)/(128 * r)
-		InvalidP {
-			display("Invalid p argument of the scrypt encryption")
-		}
-	}
-}
-
-quick_error! {
-	#[derive(Debug)]
-	pub enum SymmError wraps PrivSymmErr {
-		RustCrypto(e: rcrypto::symmetriccipher::SymmetricCipherError) {
-			display("symmetric crypto error")
-			from()
-		}
-		Ring(e: ring::error::Unspecified) {
-			display("symmetric crypto error")
-			cause(e)
-			from()
-		}
-		Offset(x: usize) {
-			display("offset {} greater than slice length", x)
-		}
-	}
-}
-
-impl SymmError {
-	pub(crate) fn offset_error(x: usize) -> SymmError {
-		SymmError(PrivSymmErr::Offset(x))
-	}
-}
-
-impl From<ring::error::Unspecified> for SymmError {
-	fn from(e: ring::error::Unspecified) -> SymmError {
-		SymmError(PrivSymmErr::Ring(e))
-	}
-}
-
-impl From<rcrypto::symmetriccipher::SymmetricCipherError> for SymmError {
-	fn from(e: rcrypto::symmetriccipher::SymmetricCipherError) -> SymmError {
-		SymmError(PrivSymmErr::RustCrypto(e))
-	}
-}

ethcore/crypto/src/hmac.rs

@@ -1,88 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use digest;
-use ring::digest::{SHA256, SHA512};
-use ring::hmac::{self, SigningContext};
-use std::marker::PhantomData;
-use std::ops::Deref;
-
-/// HMAC signature.
-pub struct Signature<T>(hmac::Signature, PhantomData<T>);
-
-impl<T> Deref for Signature<T> {
-	type Target = [u8];
-	fn deref(&self) -> &Self::Target {
-		self.0.as_ref()
-	}
-}
-
-/// HMAC signing key.
-pub struct SigKey<T>(hmac::SigningKey, PhantomData<T>);
-
-impl SigKey<digest::Sha256> {
-	pub fn sha256(key: &[u8]) -> SigKey<digest::Sha256> {
-		SigKey(hmac::SigningKey::new(&SHA256, key), PhantomData)
-	}
-}
-
-impl SigKey<digest::Sha512> {
-	pub fn sha512(key: &[u8]) -> SigKey<digest::Sha512> {
-		SigKey(hmac::SigningKey::new(&SHA512, key), PhantomData)
-	}
-}
-
-/// Compute HMAC signature of `data`.
-pub fn sign<T>(k: &SigKey<T>, data: &[u8]) -> Signature<T> {
-	Signature(hmac::sign(&k.0, data), PhantomData)
-}
-
-/// Stateful HMAC computation.
-pub struct Signer<T>(SigningContext, PhantomData<T>);
-
-impl<T> Signer<T> {
-	pub fn with(key: &SigKey<T>) -> Signer<T> {
-		Signer(hmac::SigningContext::with_key(&key.0), PhantomData)
-	}
-
-	pub fn update(&mut self, data: &[u8]) {
-		self.0.update(data)
-	}
-
-	pub fn sign(self) -> Signature<T> {
-		Signature(self.0.sign(), PhantomData)
-	}
-}
-
-/// HMAC signature verification key.
-pub struct VerifyKey<T>(hmac::VerificationKey, PhantomData<T>);
-
-impl VerifyKey<digest::Sha256> {
-	pub fn sha256(key: &[u8]) -> VerifyKey<digest::Sha256> {
-		VerifyKey(hmac::VerificationKey::new(&SHA256, key), PhantomData)
-	}
-}
-
-impl VerifyKey<digest::Sha512> {
-	pub fn sha512(key: &[u8]) -> VerifyKey<digest::Sha512> {
-		VerifyKey(hmac::VerificationKey::new(&SHA512, key), PhantomData)
-	}
-}
-
-/// Verify HMAC signature of `data`.
-pub fn verify<T>(k: &VerifyKey<T>, data: &[u8], sig: &[u8]) -> bool {
-	hmac::verify(&k.0, data, sig).is_ok()
-}

ethcore/crypto/src/lib.rs

@@ -1,76 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-//! Crypto utils used ethstore and network.
-
-extern crate crypto as rcrypto;
-extern crate ethereum_types;
-#[macro_use]
-extern crate quick_error;
-extern crate ring;
-extern crate tiny_keccak;
-
-pub mod aes;
-pub mod aes_gcm;
-pub mod error;
-pub mod scrypt;
-pub mod digest;
-pub mod hmac;
-pub mod pbkdf2;
-
-pub use error::Error;
-
-use tiny_keccak::Keccak;
-
-pub const KEY_LENGTH: usize = 32;
-pub const KEY_ITERATIONS: usize = 10240;
-pub const KEY_LENGTH_AES: usize = KEY_LENGTH / 2;
-
-/// Default authenticated data to use (in RPC).
-pub const DEFAULT_MAC: [u8; 2] = [0, 0];
-
-pub trait Keccak256<T> {
-	fn keccak256(&self) -> T where T: Sized;
-}
-
-impl<T> Keccak256<[u8; 32]> for T where T: AsRef<[u8]> {
-	fn keccak256(&self) -> [u8; 32] {
-		let mut keccak = Keccak::new_keccak256();
-		let mut result = [0u8; 32];
-		keccak.update(self.as_ref());
-		keccak.finalize(&mut result);
-		result
-	}
-}
-
-pub fn derive_key_iterations(password: &str, salt: &[u8; 32], c: u32) -> (Vec<u8>, Vec<u8>) {
-	let mut derived_key = [0u8; KEY_LENGTH];
-	pbkdf2::sha256(c, pbkdf2::Salt(salt), pbkdf2::Secret(password.as_bytes()), &mut derived_key);
-	let derived_right_bits = &derived_key[0..KEY_LENGTH_AES];
-	let derived_left_bits = &derived_key[KEY_LENGTH_AES..KEY_LENGTH];
-	(derived_right_bits.to_vec(), derived_left_bits.to_vec())
-}
-
-pub fn derive_mac(derived_left_bits: &[u8], cipher_text: &[u8]) -> Vec<u8> {
-	let mut mac = vec![0u8; KEY_LENGTH_AES + cipher_text.len()];
-	mac[0..KEY_LENGTH_AES].copy_from_slice(derived_left_bits);
-	mac[KEY_LENGTH_AES..cipher_text.len() + KEY_LENGTH_AES].copy_from_slice(cipher_text);
-	mac
-}
-
-pub fn is_equal(a: &[u8], b: &[u8]) -> bool {
-	ring::constant_time::verify_slices_are_equal(a, b).is_ok()
-}

ethcore/crypto/src/pbkdf2.rs

@@ -1,28 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use ring;
-
-pub struct Salt<'a>(pub &'a [u8]);
-pub struct Secret<'a>(pub &'a [u8]);
-
-pub fn sha256(iter: u32, salt: Salt, sec: Secret, out: &mut [u8; 32]) {
-	ring::pbkdf2::derive(&ring::digest::SHA256, iter, salt.0, sec.0, &mut out[..])
-}
-
-pub fn sha512(iter: u32, salt: Salt, sec: Secret, out: &mut [u8; 64]) {
-	ring::pbkdf2::derive(&ring::digest::SHA512, iter, salt.0, sec.0, &mut out[..])
-}

ethcore/crypto/src/scrypt.rs

@@ -1,38 +0,0 @@
-// Copyright 2015-2018 Parity Technologies (UK) Ltd.
-// This file is part of Parity.
-
-// Parity is free software: you can redistribute it and/or modify
-// it under the terms of the GNU General Public License as published by
-// the Free Software Foundation, either version 3 of the License, or
-// (at your option) any later version.
-
-// Parity is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-// GNU General Public License for more details.
-
-// You should have received a copy of the GNU General Public License
-// along with Parity.  If not, see <http://www.gnu.org/licenses/>.
-
-use error::ScryptError;
-use rcrypto::scrypt::{scrypt, ScryptParams};
-use super::{KEY_LENGTH_AES, KEY_LENGTH};
-
-pub fn derive_key(pass: &str, salt: &[u8; 32], n: u32, p: u32, r: u32) -> Result<(Vec<u8>, Vec<u8>), ScryptError> {
-	// sanity checks
-	let log_n = (32 - n.leading_zeros() - 1) as u8;
-	if log_n as u32 >= r * 16 {
-		return Err(ScryptError::InvalidN);
-	}
-
-	if p as u64 > ((u32::max_value() as u64 - 1) * 32)/(128 * (r as u64)) {
-		return Err(ScryptError::InvalidP);
-	}
-
-	let mut derived_key = vec![0u8; KEY_LENGTH];
-	let scrypt_params = ScryptParams::new(log_n, r, p);
-	scrypt(pass.as_bytes(), salt, &scrypt_params, &mut derived_key);
-	let derived_right_bits = &derived_key[0..KEY_LENGTH_AES];
-	let derived_left_bits = &derived_key[KEY_LENGTH_AES..KEY_LENGTH];
-	Ok((derived_right_bits.to_vec(), derived_left_bits.to_vec()))
-}

ethcore/evm/Cargo.toml

@@ -10,8 +10,8 @@ heapsize = "0.4"
 lazy_static = "1.0"
 log = "0.3"
 vm = { path = "../vm" }
-keccak-hash = { path = "../../util/hash" }
-parking_lot = "0.5"
+keccak-hash = { git = "https://github.com/paritytech/parity-common" }
+parking_lot = "0.6"
 memory-cache = { path = "../../util/memory_cache" }
 
 [dev-dependencies]

ethcore/evm/src/interpreter/gasometer.rs

@@ -113,7 +113,7 @@ impl<Gas: evm::CostType> Gasometer<Gas> {
 		current_mem_size: usize,
 	) -> vm::Result<InstructionRequirements<Gas>> {
 		let schedule = ext.schedule();
-		let tier = instructions::get_tier_idx(info.tier);
+		let tier = info.tier.idx();
 		let default_gas = Gas::from(schedule.tier_step_gas[tier]);
 
 		let cost = match instruction {
@@ -179,8 +179,8 @@ impl<Gas: evm::CostType> Gasometer<Gas> {
 			instructions::EXTCODECOPY => {
 				Request::GasMemCopy(schedule.extcodecopy_base_gas.into(), mem_needed(stack.peek(1), stack.peek(3))?, Gas::from_u256(*stack.peek(3))?)
 			},
-			instructions::LOG0...instructions::LOG4 => {
-				let no_of_topics = instructions::get_log_topics(instruction);
+			instructions::LOG0 | instructions::LOG1 | instructions::LOG2 | instructions::LOG3 | instructions::LOG4 => {
+				let no_of_topics = instruction.log_topics().expect("log_topics always return some for LOG* instructions; qed");
 				let log_gas = schedule.log_gas + schedule.log_topic_gas * no_of_topics;
 
 				let data_gas = overflowing!(Gas::from_u256(*stack.peek(1))?.overflow_mul(Gas::from(schedule.log_data_gas)));

ethcore/evm/src/interpreter/informant.rs

@@ -39,7 +39,7 @@ mod inner {
 	use std::collections::HashMap;
 	use std::time::{Instant, Duration};
 
-	use bigint::prelude::U256;
+	use ethereum_types::U256;
 
 	use interpreter::stack::Stack;
 	use instructions::{Instruction, InstructionInfo, INSTRUCTIONS};

ethcore/evm/src/interpreter/mod.rs

@@ -64,7 +64,6 @@ struct CodeReader<'a> {
 }
 
 impl<'a> CodeReader<'a> {
-
 	/// Create new code reader - starting at position 0.
 	fn new(code: &'a [u8]) -> Self {
 		CodeReader {
@@ -81,7 +80,7 @@ impl<'a> CodeReader<'a> {
 		U256::from(&self.code[pos..max])
 	}
 
-	fn len (&self) -> usize {
+	fn len(&self) -> usize {
 		self.code.len()
 	}
 }
@@ -124,24 +123,31 @@ impl<Cost: CostType> vm::Vm for Interpreter<Cost> {
 		let mut gasometer = Gasometer::<Cost>::new(Cost::from_u256(params.gas)?);
 		let mut stack = VecStack::with_capacity(ext.schedule().stack_limit, U256::zero());
 		let mut reader = CodeReader::new(code);
-		let infos = &*instructions::INSTRUCTIONS;
 
 		while reader.position < code.len() {
-			let instruction = code[reader.position];
+			let opcode = code[reader.position];
+			let instruction = Instruction::from_u8(opcode);
 			reader.position += 1;
 
 			// TODO: make compile-time removable if too much of a performance hit.
 			do_trace = do_trace && ext.trace_next_instruction(
-				reader.position - 1, instruction, gasometer.current_gas.as_u256(),
+				reader.position - 1, opcode, gasometer.current_gas.as_u256(),
 			);
 
-			let info = &infos[instruction as usize];
+			if instruction.is_none() {
+				return Err(vm::Error::BadInstruction {
+					instruction: opcode
+				});
+			}
+			let instruction = instruction.expect("None case is checked above; qed");
+
+			let info = instruction.info();
 			self.verify_instruction(ext, instruction, info, &stack)?;
 
 			// Calculate gas cost
 			let requirements = gasometer.requirements(ext, instruction, info, &stack, self.mem.size())?;
 			if do_trace {
-				ext.trace_prepare_execute(reader.position - 1, instruction, requirements.gas_cost.as_u256());
+				ext.trace_prepare_execute(reader.position - 1, opcode, requirements.gas_cost.as_u256());
 			}
 
 			gasometer.verify_gas(&requirements.gas_cost)?;
@@ -227,13 +233,7 @@ impl<Cost: CostType> Interpreter<Cost> {
 			((instruction == instructions::SHL || instruction == instructions::SHR || instruction == instructions::SAR) && !schedule.have_bitwise_shifting) {
 
 			return Err(vm::Error::BadInstruction {
-				instruction: instruction
-			});
-		}
-
-		if info.tier == instructions::GasPriceTier::Invalid {
-			return Err(vm::Error::BadInstruction {
-				instruction: instruction
+				instruction: instruction as u8
 			});
 		}
 
@@ -396,7 +396,7 @@ impl<Cost: CostType> Interpreter<Cost> {
 					},
 					instructions::DELEGATECALL => (&params.sender, &params.address, true, CallType::DelegateCall),
 					instructions::STATICCALL => (&params.address, &code_address, true, CallType::StaticCall),
-					_ => panic!(format!("Unexpected instruction {} in CALL branch.", instruction))
+					_ => panic!(format!("Unexpected instruction {:?} in CALL branch.", instruction))
 				};
 
 				// clear return data buffer before creating new call frame.
@@ -453,8 +453,8 @@ impl<Cost: CostType> Interpreter<Cost> {
 				ext.suicide(&u256_to_address(&address))?;
 				return Ok(InstructionResult::StopExecution);
 			},
-			instructions::LOG0...instructions::LOG4 => {
-				let no_of_topics = instructions::get_log_topics(instruction);
+			instructions::LOG0 | instructions::LOG1 | instructions::LOG2 | instructions::LOG3 | instructions::LOG4 => {
+				let no_of_topics = instruction.log_topics().expect("log_topics always return some for LOG* instructions; qed");
 
 				let offset = stack.pop_back();
 				let size = stack.pop_back();
@@ -464,8 +464,15 @@ impl<Cost: CostType> Interpreter<Cost> {
 					.collect();
 				ext.log(topics, self.mem.read_slice(offset, size))?;
 			},
-			instructions::PUSH1...instructions::PUSH32 => {
-				let bytes = instructions::get_push_bytes(instruction);
+			instructions::PUSH1 | instructions::PUSH2 | instructions::PUSH3 | instructions::PUSH4 |
+			instructions::PUSH5 | instructions::PUSH6 | instructions::PUSH7 | instructions::PUSH8 |
+			instructions::PUSH9 | instructions::PUSH10 | instructions::PUSH11 | instructions::PUSH12 |
+			instructions::PUSH13 | instructions::PUSH14 | instructions::PUSH15 | instructions::PUSH16 |
+			instructions::PUSH17 | instructions::PUSH18 | instructions::PUSH19 | instructions::PUSH20 |
+			instructions::PUSH21 | instructions::PUSH22 | instructions::PUSH23 | instructions::PUSH24 |
+			instructions::PUSH25 | instructions::PUSH26 | instructions::PUSH27 | instructions::PUSH28 |
+			instructions::PUSH29 | instructions::PUSH30 | instructions::PUSH31 | instructions::PUSH32 => {
+				let bytes = instruction.push_bytes().expect("push_bytes always return some for PUSH* instructions");
 				let val = code.read(bytes);
 				stack.push(val);
 			},
@@ -609,73 +616,22 @@ impl<Cost: CostType> Interpreter<Cost> {
 			instructions::GASLIMIT => {
 				stack.push(ext.env_info().gas_limit.clone());
 			},
-			_ => {
-				self.exec_stack_instruction(instruction, stack)?;
-			}
-		};
-		Ok(InstructionResult::Ok)
-	}
 
-	fn copy_data_to_memory(mem: &mut Vec<u8>, stack: &mut Stack<U256>, source: &[u8]) {
-		let dest_offset = stack.pop_back();
-		let source_offset = stack.pop_back();
-		let size = stack.pop_back();
-		let source_size = U256::from(source.len());
+			// Stack instructions
 
-		let output_end = match source_offset > source_size || size > source_size || source_offset + size > source_size {
-			true => {
-				let zero_slice = if source_offset > source_size {
-					mem.writeable_slice(dest_offset, size)
-				} else {
-					mem.writeable_slice(dest_offset + source_size - source_offset, source_offset + size - source_size)
-				};
-				for i in zero_slice.iter_mut() {
-					*i = 0;
-				}
-				source.len()
-			},
-			false => (size.low_u64() + source_offset.low_u64()) as usize
-		};
-
-		if source_offset < source_size {
-			let output_begin = source_offset.low_u64() as usize;
-			mem.write_slice(dest_offset, &source[output_begin..output_end]);
-		}
-	}
-
-	fn verify_jump(&self, jump_u: U256, valid_jump_destinations: &BitSet) -> vm::Result<usize> {
-		let jump = jump_u.low_u64() as usize;
-
-		if valid_jump_destinations.contains(jump) && U256::from(jump) == jump_u {
-			Ok(jump)
-		} else {
-			Err(vm::Error::BadJumpDestination {
-				destination: jump
-			})
-		}
-	}
-
-	fn is_zero(&self, val: &U256) -> bool {
-		val.is_zero()
-	}
-
-	fn bool_to_u256(&self, val: bool) -> U256 {
-		if val {
-			U256::one()
-		} else {
-			U256::zero()
-		}
-	}
-
-	fn exec_stack_instruction(&self, instruction: Instruction, stack: &mut Stack<U256>) -> vm::Result<()> {
-		match instruction {
-			instructions::DUP1...instructions::DUP16 => {
-				let position = instructions::get_dup_position(instruction);
+			instructions::DUP1 | instructions::DUP2 | instructions::DUP3 | instructions::DUP4 |
+			instructions::DUP5 | instructions::DUP6 | instructions::DUP7 | instructions::DUP8 |
+			instructions::DUP9 | instructions::DUP10 | instructions::DUP11 | instructions::DUP12 |
+			instructions::DUP13 | instructions::DUP14 | instructions::DUP15 | instructions::DUP16 => {
+				let position = instruction.dup_position().expect("dup_position always return some for DUP* instructions");
 				let val = stack.peek(position).clone();
 				stack.push(val);
 			},
-			instructions::SWAP1...instructions::SWAP16 => {
-				let position = instructions::get_swap_position(instruction);
+			instructions::SWAP1 | instructions::SWAP2 | instructions::SWAP3 | instructions::SWAP4 |
+			instructions::SWAP5 | instructions::SWAP6 | instructions::SWAP7 | instructions::SWAP8 |
+			instructions::SWAP9 | instructions::SWAP10 | instructions::SWAP11 | instructions::SWAP12 |
+			instructions::SWAP13 | instructions::SWAP14 | instructions::SWAP15 | instructions::SWAP16 => {
+				let position = instruction.swap_position().expect("swap_position always return some for SWAP* instructions");
 				stack.swap_with_top(position)
 			},
 			instructions::POP => {
@@ -923,15 +879,60 @@ impl<Cost: CostType> Interpreter<Cost> {
 				};
 				stack.push(result);
 			},
-			_ => {
-				return Err(vm::Error::BadInstruction {
-					instruction: instruction
-				});
-			}
-		}
-		Ok(())
+		};
+		Ok(InstructionResult::Ok)
 	}
 
+	fn copy_data_to_memory(mem: &mut Vec<u8>, stack: &mut Stack<U256>, source: &[u8]) {
+		let dest_offset = stack.pop_back();
+		let source_offset = stack.pop_back();
+		let size = stack.pop_back();
+		let source_size = U256::from(source.len());
+
+		let output_end = match source_offset > source_size || size > source_size || source_offset + size > source_size {
+			true => {
+				let zero_slice = if source_offset > source_size {
+					mem.writeable_slice(dest_offset, size)
+				} else {
+					mem.writeable_slice(dest_offset + source_size - source_offset, source_offset + size - source_size)
+				};
+				for i in zero_slice.iter_mut() {
+					*i = 0;
+				}
+				source.len()
+			},
+			false => (size.low_u64() + source_offset.low_u64()) as usize
+		};
+
+		if source_offset < source_size {
+			let output_begin = source_offset.low_u64() as usize;
+			mem.write_slice(dest_offset, &source[output_begin..output_end]);
+		}
+	}
+
+	fn verify_jump(&self, jump_u: U256, valid_jump_destinations: &BitSet) -> vm::Result<usize> {
+		let jump = jump_u.low_u64() as usize;
+
+		if valid_jump_destinations.contains(jump) && U256::from(jump) == jump_u {
+			Ok(jump)
+		} else {
+			Err(vm::Error::BadJumpDestination {
+				destination: jump
+			})
+		}
+	}
+
+	fn is_zero(&self, val: &U256) -> bool {
+		val.is_zero()
+	}
+
+	fn bool_to_u256(&self, val: bool) -> U256 {
+		if val {
+			U256::one()
+		} else {
+			U256::zero()
+		}
+	}
 }
 
 fn get_and_reset_sign(value: U256) -> (U256, bool) {

ethcore/evm/src/interpreter/shared_cache.rs

@@ -21,7 +21,7 @@ use ethereum_types::H256;
 use parking_lot::Mutex;
 use memory_cache::MemoryLruCache;
 use bit_set::BitSet;
-use super::super::instructions;
+use super::super::instructions::{self, Instruction};
 
 const DEFAULT_CACHE_SIZE: usize = 4 * 1024 * 1024;
 
@@ -70,12 +70,14 @@ impl SharedCache {
 		let mut position = 0;
 
 		while position < code.len() {
-			let instruction = code[position];
+			let instruction = Instruction::from_u8(code[position]);
 
-			if instruction == instructions::JUMPDEST {
-				jump_dests.insert(position);
-			} else if instructions::is_push(instruction) {
-				position += instructions::get_push_bytes(instruction);
+			if let Some(instruction) = instruction {
+				if instruction == instructions::JUMPDEST {
+					jump_dests.insert(position);
+				} else if let Some(push_bytes) = instruction.push_bytes() {
+					position += push_bytes;
+				}
 			}
 			position += 1;
 		}

ethcore/evm/src/lib.rs

@@ -52,6 +52,6 @@ pub use vm::{
     GasLeft, ReturnData
 };
 pub use self::evm::{Finalize, FinalizationResult, CostType};
-pub use self::instructions::{InstructionInfo, INSTRUCTIONS, push_bytes};
+pub use self::instructions::{InstructionInfo, Instruction};
 pub use self::vmtype::VMType;
 pub use self::factory::Factory;

ethcore/light/Cargo.toml

@@ -9,18 +9,19 @@ authors = ["Parity Technologies <admin@parity.io>"]
 [dependencies]
 log = "0.3"
 ethcore = { path = ".."}
-ethcore-bytes = { path = "../../util/bytes" }
+parity-bytes = { git = "https://github.com/paritytech/parity-common" }
 ethcore-transaction = { path = "../transaction" }
 ethereum-types = "0.3"
-memorydb = { path = "../../util/memorydb" }
-patricia-trie = { path = "../../util/patricia_trie" }
+memorydb = { git = "https://github.com/paritytech/parity-common" }
+patricia-trie = { git = "https://github.com/paritytech/parity-common" }
+patricia-trie-ethereum = { path = "../../util/patricia-trie-ethereum" }
 ethcore-network = { path = "../../util/network" }
 ethcore-io = { path = "../../util/io" }
-hashdb = { path = "../../util/hashdb" }
+hashdb = { git = "https://github.com/paritytech/parity-common" }
 heapsize = "0.4"
 vm = { path = "../vm" }
-plain_hasher = { path = "../../util/plain_hasher" }
-rlp = { path = "../../util/rlp" }
+plain_hasher = { git = "https://github.com/paritytech/parity-common" }
+rlp = { git = "https://github.com/paritytech/parity-common" }
 rlp_derive = { path = "../../util/rlp_derive" }
 smallvec = "0.4"
 futures = "0.1"
@@ -29,16 +30,18 @@ itertools = "0.5"
 bincode = "0.8.0"
 serde = "1.0"
 serde_derive = "1.0"
-parking_lot = "0.5"
+parking_lot = "0.6"
 stats = { path = "../../util/stats" }
-keccak-hash = { path = "../../util/hash" }
-triehash = { path = "../../util/triehash" }
-kvdb = { path = "../../util/kvdb" }
+keccak-hash = { git = "https://github.com/paritytech/parity-common" }
+keccak-hasher = { path = "../../util/keccak-hasher" }
+triehash-ethereum = { version = "0.2",  path = "../../util/triehash-ethereum" }
+kvdb = { git = "https://github.com/paritytech/parity-common" }
 memory-cache = { path = "../../util/memory_cache" }
-error-chain = { version = "0.11", default-features = false }
+error-chain = { version = "0.12", default-features = false }
 
 [dev-dependencies]
-kvdb-memorydb = { path = "../../util/kvdb-memorydb" }
+ethcore = { path = "..", features = ["test-helpers"] }
+kvdb-memorydb = { git = "https://github.com/paritytech/parity-common" }
 tempdir = "0.3"
 
 [features]

ethcore/light/src/cht.rs

@@ -26,9 +26,11 @@
 use ethcore::ids::BlockId;
 use ethereum_types::{H256, U256};
 use hashdb::HashDB;
+use keccak_hasher::KeccakHasher;
 use memorydb::MemoryDB;
 use bytes::Bytes;
-use trie::{self, TrieMut, TrieDBMut, Trie, TrieDB, Recorder};
+use trie::{TrieMut, Trie, Recorder};
+use ethtrie::{self, TrieDB, TrieDBMut};
 use rlp::{RlpStream, Rlp};
 
 // encode a key.
@@ -50,13 +52,13 @@ pub const SIZE: u64 = 2048;
 /// A canonical hash trie. This is generic over any database it can query.
 /// See module docs for more details.
 #[derive(Debug, Clone)]
-pub struct CHT<DB: HashDB> {
+pub struct CHT<DB: HashDB<KeccakHasher>> {
 	db: DB,
 	root: H256, // the root of this CHT.
 	number: u64,
 }
 
-impl<DB: HashDB> CHT<DB> {
+impl<DB: HashDB<KeccakHasher>> CHT<DB> {
 	/// Query the root of the CHT.
 	pub fn root(&self) -> H256 { self.root }
 
@@ -66,7 +68,7 @@ impl<DB: HashDB> CHT<DB> {
 	/// Generate an inclusion proof for the entry at a specific block.
 	/// Nodes before level `from_level` will be omitted.
 	/// Returns an error on an incomplete trie, and `Ok(None)` on an unprovable request.
-	pub fn prove(&self, num: u64, from_level: u32) -> trie::Result<Option<Vec<Bytes>>> {
+	pub fn prove(&self, num: u64, from_level: u32) -> ethtrie::Result<Option<Vec<Bytes>>> {
 		if block_to_cht_number(num) != Some(self.number) { return Ok(None) }
 
 		let mut recorder = Recorder::with_depth(from_level);
@@ -90,10 +92,10 @@ pub struct BlockInfo {
 /// Build an in-memory CHT from a closure which provides necessary information
 /// about blocks. If the fetcher ever fails to provide the info, the CHT
 /// will not be generated.
-pub fn build<F>(cht_num: u64, mut fetcher: F) ->  Option<CHT<MemoryDB>>
+pub fn build<F>(cht_num: u64, mut fetcher: F) ->  Option<CHT<MemoryDB<KeccakHasher>>>
 	where F: FnMut(BlockId) -> Option<BlockInfo>
 {
-	let mut db = MemoryDB::new();
+	let mut db = MemoryDB::<KeccakHasher>::new();
 
 	// start from the last block by number and work backwards.
 	let last_num = start_number(cht_num + 1) - 1;
@@ -147,7 +149,7 @@ pub fn compute_root<I>(cht_num: u64, iterable: I) -> Option<H256>
 /// verify the given trie branch and extract the canonical hash and total difficulty.
 // TODO: better support for partially-checked queries.
 pub fn check_proof(proof: &[Bytes], num: u64, root: H256) -> Option<(H256, U256)> {
-	let mut db = MemoryDB::new();
+	let mut db = MemoryDB::<KeccakHasher>::new();
 
 	for node in proof { db.insert(&node[..]); }
 	let res = match TrieDB::new(&db, &root) {

ethcore/light/src/client/header_chain.rs

@@ -28,28 +28,21 @@
 use std::collections::BTreeMap;
 use std::sync::Arc;
 
+use cache::Cache;
 use cht;
-
 use ethcore::block_status::BlockStatus;
-use ethcore::error::{Error, ErrorKind, BlockImportError, BlockImportErrorKind, BlockError};
 use ethcore::encoded;
+use ethcore::engines::epoch::{Transition as EpochTransition, PendingTransition as PendingEpochTransition};
+use ethcore::error::{Error, BlockImportError, BlockImportErrorKind, BlockError};
 use ethcore::header::Header;
 use ethcore::ids::BlockId;
 use ethcore::spec::{Spec, SpecHardcodedSync};
-use ethcore::engines::epoch::{
-	Transition as EpochTransition,
-	PendingTransition as PendingEpochTransition
-};
-
-use rlp::{Encodable, Decodable, DecoderError, RlpStream, Rlp};
-use heapsize::HeapSizeOf;
 use ethereum_types::{H256, H264, U256};
-use plain_hasher::H256FastMap;
+use heapsize::HeapSizeOf;
 use kvdb::{DBTransaction, KeyValueDB};
-
-use cache::Cache;
 use parking_lot::{Mutex, RwLock};
-
+use plain_hasher::H256FastMap;
+use rlp::{Encodable, Decodable, DecoderError, RlpStream, Rlp};
 use smallvec::SmallVec;
 
 /// Store at least this many candidate headers at all times.
@@ -260,7 +253,7 @@ impl HeaderChain {
 			let best_block = {
 				let era = match candidates.get(&curr.best_num) {
 					Some(era) => era,
-					None => bail!(ErrorKind::Database("Database corrupt: highest block referenced but no data.".into())),
+					None => bail!("Database corrupt: highest block referenced but no data."),
 				};
 
 				let best = &era.candidates[0];
@@ -583,7 +576,7 @@ impl HeaderChain {
 					} else {
 						let msg = format!("header of block #{} not found in DB ; database in an \
 											inconsistent state", h_num);
-						bail!(ErrorKind::Database(msg.into()));
+						bail!(msg);
 					};
 
 					let decoded = header.decode().expect("decoding db value failed");
@@ -591,9 +584,8 @@ impl HeaderChain {
 					let entry: Entry = {
 						let bytes = self.db.get(self.col, era_key(h_num).as_bytes())?
 							.ok_or_else(|| {
-								let msg = format!("entry for era #{} not found in DB ; database \
-													in an inconsistent state", h_num);
-								ErrorKind::Database(msg.into())
+								format!("entry for era #{} not found in DB ; database \
+										in an inconsistent state", h_num)
 							})?;
 						::rlp::decode(&bytes).expect("decoding db value failed")
 					};
@@ -601,9 +593,8 @@ impl HeaderChain {
 					let total_difficulty = entry.candidates.iter()
 						.find(|c| c.hash == decoded.hash())
 						.ok_or_else(|| {
-							let msg = "no candidate matching block found in DB ; database in an \
-										inconsistent state";
-							ErrorKind::Database(msg.into())
+							"no candidate matching block found in DB ; database in an \
+										inconsistent state"
 						})?
 						.total_difficulty;
 

ethcore/light/src/client/service.rs

@@ -21,11 +21,10 @@ use std::fmt;
 use std::sync::Arc;
 
 use ethcore::client::ClientIoMessage;
-use ethcore::db;
+use ethcore::{db, BlockChainDB};
 use ethcore::error::Error as CoreError;
 use ethcore::spec::Spec;
 use io::{IoContext, IoError, IoHandler, IoService};
-use kvdb::KeyValueDB;
 
 use cache::Cache;
 use parking_lot::Mutex;
@@ -65,11 +64,10 @@ pub struct Service<T> {
 
 impl<T: ChainDataFetcher> Service<T> {
 	/// Start the service: initialize I/O workers and client itself.
-	pub fn start(config: ClientConfig, spec: &Spec, fetcher: T, db: Arc<KeyValueDB>, cache: Arc<Mutex<Cache>>) -> Result<Self, Error> {
-
+	pub fn start(config: ClientConfig, spec: &Spec, fetcher: T, db: Arc<BlockChainDB>, cache: Arc<Mutex<Cache>>) -> Result<Self, Error> {
 		let io_service = IoService::<ClientIoMessage>::start().map_err(Error::Io)?;
 		let client = Arc::new(Client::new(config,
-			db,
+			db.key_value().clone(),
 			db::COL_LIGHT_CHAIN,
 			spec,
 			fetcher,
@@ -122,12 +120,11 @@ mod tests {
 	use client::fetch;
 	use std::time::Duration;
 	use parking_lot::Mutex;
-	use kvdb_memorydb;
-	use ethcore::db::NUM_COLUMNS;
+	use ethcore::test_helpers;
 
 	#[test]
 	fn it_works() {
-		let db = Arc::new(kvdb_memorydb::create(NUM_COLUMNS.unwrap_or(0)));
+		let db = test_helpers::new_db();
 		let spec = Spec::new_test();
 		let cache = Arc::new(Mutex::new(Cache::new(Default::default(), Duration::from_secs(6 * 3600))));
 

ethcore/light/src/lib.rs

@@ -43,7 +43,7 @@ pub mod provider;
 mod types;
 
 pub use self::cache::Cache;
-pub use self::provider::Provider;
+pub use self::provider::{Provider, MAX_HEADERS_PER_REQUEST};
 pub use self::transaction_queue::TransactionQueue;
 pub use types::request as request;
 
@@ -56,7 +56,7 @@ extern crate log;
 extern crate bincode;
 extern crate ethcore_io as io;
 extern crate ethcore_network as network;
-extern crate ethcore_bytes as bytes;
+extern crate parity_bytes as bytes;
 extern crate ethcore_transaction as transaction;
 extern crate ethereum_types;
 extern crate ethcore;
@@ -64,8 +64,10 @@ extern crate hashdb;
 extern crate heapsize;
 extern crate futures;
 extern crate itertools;
+extern crate keccak_hasher;
 extern crate memorydb;
 extern crate patricia_trie as trie;
+extern crate patricia_trie_ethereum as ethtrie;
 extern crate plain_hasher;
 extern crate rand;
 extern crate rlp;
@@ -77,7 +79,7 @@ extern crate smallvec;
 extern crate stats;
 extern crate vm;
 extern crate keccak_hash as hash;
-extern crate triehash;
+extern crate triehash_ethereum as triehash;
 extern crate kvdb;
 extern crate memory_cache;
 #[macro_use]

ethcore/light/src/net/mod.rs

@@ -20,22 +20,20 @@
 
 use transaction::UnverifiedTransaction;
 
-use io::TimerToken;
-use network::{NetworkProtocolHandler, NetworkContext, PeerId};
-use rlp::{RlpStream, Rlp};
 use ethereum_types::{H256, U256};
+use io::TimerToken;
 use kvdb::DBValue;
+use network::{NetworkProtocolHandler, NetworkContext, PeerId};
 use parking_lot::{Mutex, RwLock};
-use std::time::{Duration, Instant};
-
-use std::collections::{HashMap, HashSet};
-use std::fmt;
-use std::sync::Arc;
-use std::sync::atomic::{AtomicUsize, Ordering};
-use std::ops::{BitOr, BitAnd, Not};
-
 use provider::Provider;
 use request::{Request, NetworkRequests as Requests, Response};
+use rlp::{RlpStream, Rlp};
+use std::collections::{HashMap, HashSet};
+use std::fmt;
+use std::ops::{BitOr, BitAnd, Not};
+use std::sync::Arc;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use std::time::{Duration, Instant};
 
 use self::request_credits::{Credits, FlowParams};
 use self::context::{Ctx, TickCtx};
@@ -647,7 +645,7 @@ impl LightProtocol {
 	fn propagate_transactions(&self, io: &IoContext) {
 		if self.capabilities.read().tx_relay { return }
 
-		let ready_transactions = self.provider.ready_transactions();
+		let ready_transactions = self.provider.transactions_to_propagate();
 		if ready_transactions.is_empty() { return }
 
 		trace!(target: "pip", "propagate transactions: {} ready", ready_transactions.len());

ethcore/light/src/net/status.rs

@@ -16,8 +16,8 @@
 
 //! Peer status and capabilities.
 
-use rlp::{DecoderError, Encodable, Decodable, RlpStream, Rlp};
 use ethereum_types::{H256, U256};
+use rlp::{DecoderError, Encodable, Decodable, RlpStream, Rlp};
 
 use super::request_credits::FlowParams;
 

ethcore/light/src/net/tests/mod.rs

@@ -19,20 +19,18 @@
 
 use ethcore::blockchain_info::BlockChainInfo;
 use ethcore::client::{EachBlockWith, TestBlockChainClient};
-use ethcore::ids::BlockId;
 use ethcore::encoded;
-use network::{PeerId, NodeId};
-use transaction::{Action, PendingTransaction};
-
+use ethcore::ids::BlockId;
+use ethereum_types::{H256, U256, Address};
+use net::{LightProtocol, Params, packet, Peer};
 use net::context::IoContext;
 use net::status::{Capabilities, Status};
-use net::{LightProtocol, Params, packet, Peer};
+use network::{PeerId, NodeId};
 use provider::Provider;
 use request;
 use request::*;
-
 use rlp::{Rlp, RlpStream};
-use ethereum_types::{H256, U256, Address};
+use transaction::{Action, PendingTransaction};
 
 use std::sync::Arc;
 use std::time::Instant;
@@ -173,8 +171,8 @@ impl Provider for TestProvider {
 		})
 	}
 
-	fn ready_transactions(&self) -> Vec<PendingTransaction> {
-		self.0.client.ready_transactions()
+	fn transactions_to_propagate(&self) -> Vec<PendingTransaction> {
+		self.0.client.transactions_to_propagate()
 	}
 }
 

ethcore/light/src/on_demand/mod.rs

@@ -204,6 +204,8 @@ fn guess_capabilities(requests: &[CheckedRequest]) -> Capabilities {
 				caps.serve_headers = true,
 			CheckedRequest::HeaderByHash(_, _) =>
 				caps.serve_headers = true,
+			CheckedRequest::HeaderWithAncestors(_, _) =>
+				caps.serve_headers = true,
 			CheckedRequest::TransactionIndex(_, _) => {} // hashes yield no info.
 			CheckedRequest::Signal(_, _) =>
 				caps.serve_headers = true,

ethcore/light/src/on_demand/request.rs

@@ -16,28 +16,28 @@
 
 //! Request types, verification, and verification errors.
 
+use std::cmp;
 use std::sync::Arc;
 
+use bytes::Bytes;
 use ethcore::basic_account::BasicAccount;
 use ethcore::encoded;
 use ethcore::engines::{EthEngine, StateDependentProof};
 use ethcore::machine::EthereumMachine;
 use ethcore::receipt::Receipt;
 use ethcore::state::{self, ProvedExecution};
-use transaction::SignedTransaction;
-use vm::EnvInfo;
-use hash::{KECCAK_NULL_RLP, KECCAK_EMPTY, KECCAK_EMPTY_LIST_RLP, keccak};
-
-use request::{self as net_request, IncompleteRequest, CompleteRequest, Output, OutputKind, Field};
-
-use rlp::{RlpStream, Rlp};
 use ethereum_types::{H256, U256, Address};
-use parking_lot::Mutex;
+use ethtrie::{TrieError, TrieDB};
+use hash::{KECCAK_NULL_RLP, KECCAK_EMPTY, KECCAK_EMPTY_LIST_RLP, keccak};
 use hashdb::HashDB;
 use kvdb::DBValue;
-use bytes::Bytes;
 use memorydb::MemoryDB;
-use trie::{Trie, TrieDB, TrieError};
+use parking_lot::Mutex;
+use request::{self as net_request, IncompleteRequest, CompleteRequest, Output, OutputKind, Field};
+use rlp::{RlpStream, Rlp};
+use transaction::SignedTransaction;
+use trie::Trie;
+use vm::EnvInfo;
 
 const SUPPLIED_MATCHES: &'static str = "supplied responses always match produced requests; enforced by `check_response`; qed";
 
@@ -48,6 +48,8 @@ pub enum Request {
 	HeaderProof(HeaderProof),
 	/// A request for a header by hash.
 	HeaderByHash(HeaderByHash),
+	/// A request for a header by hash with a range of its ancestors.
+	HeaderWithAncestors(HeaderWithAncestors),
 	/// A request for the index of a transaction.
 	TransactionIndex(TransactionIndex),
 	/// A request for block receipts.
@@ -137,6 +139,7 @@ macro_rules! impl_single {
 // implement traits for each kind of request.
 impl_single!(HeaderProof, HeaderProof, (H256, U256));
 impl_single!(HeaderByHash, HeaderByHash, encoded::Header);
+impl_single!(HeaderWithAncestors, HeaderWithAncestors, Vec<encoded::Header>);
 impl_single!(TransactionIndex, TransactionIndex, net_request::TransactionIndexResponse);
 impl_single!(Receipts, BlockReceipts, Vec<Receipt>);
 impl_single!(Body, Body, encoded::Block);
@@ -247,6 +250,7 @@ impl From<encoded::Header> for HeaderRef {
 pub enum CheckedRequest {
 	HeaderProof(HeaderProof, net_request::IncompleteHeaderProofRequest),
 	HeaderByHash(HeaderByHash, net_request::IncompleteHeadersRequest),
+	HeaderWithAncestors(HeaderWithAncestors, net_request::IncompleteHeadersRequest),
 	TransactionIndex(TransactionIndex, net_request::IncompleteTransactionIndexRequest),
 	Receipts(BlockReceipts, net_request::IncompleteReceiptsRequest),
 	Body(Body, net_request::IncompleteBodyRequest),
@@ -268,6 +272,16 @@ impl From<Request> for CheckedRequest {
 				};
 				CheckedRequest::HeaderByHash(req, net_req)
 			}
+			Request::HeaderWithAncestors(req) => {
+				let net_req = net_request::IncompleteHeadersRequest {
+					start: req.block_hash.map(Into::into),
+					skip: 0,
+					max: req.ancestor_count + 1,
+					reverse: true,
+				};
+				trace!(target: "on_demand", "HeaderWithAncestors Request, {:?}", net_req);
+				CheckedRequest::HeaderWithAncestors(req, net_req)
+			}
 			Request::HeaderProof(req) => {
 				let net_req = net_request::IncompleteHeaderProofRequest {
 					num: req.num().into(),
@@ -336,6 +350,7 @@ impl CheckedRequest {
 		match self {
 			CheckedRequest::HeaderProof(_, req) => NetRequest::HeaderProof(req),
 			CheckedRequest::HeaderByHash(_, req) => NetRequest::Headers(req),
+			CheckedRequest::HeaderWithAncestors(_, req) => NetRequest::Headers(req),
 			CheckedRequest::TransactionIndex(_, req) => NetRequest::TransactionIndex(req),
 			CheckedRequest::Receipts(_, req) => NetRequest::Receipts(req),
 			CheckedRequest::Body(_, req) => NetRequest::Body(req),
@@ -391,6 +406,27 @@ impl CheckedRequest {
 
 				None
 			}
+			CheckedRequest::HeaderWithAncestors(_, ref req) => {
+				if req.skip != 1 || !req.reverse {
+					return None;
+				}
+
+				if let Some(&net_request::HashOrNumber::Hash(start)) = req.start.as_ref() {
+					let mut result = Vec::with_capacity(req.max as usize);
+					let mut hash = start;
+					let mut cache = cache.lock();
+					for _ in 0..req.max {
+						match cache.block_header(&hash) {
+							Some(header) => {
+								hash = header.parent_hash();
+								result.push(header);
+							}
+							None => return None,
+						}
+					}
+					Some(Response::HeaderWithAncestors(result))
+				} else { None }
+			}
 			CheckedRequest::Receipts(ref check, ref req) => {
 				// empty transactions -> no receipts
 				if check.0.as_ref().ok().map_or(false, |hdr| hdr.receipts_root() == KECCAK_NULL_RLP) {
@@ -459,6 +495,7 @@ macro_rules! match_me {
 		match $me {
 			CheckedRequest::HeaderProof($check, $req) => $e,
 			CheckedRequest::HeaderByHash($check, $req) => $e,
+			CheckedRequest::HeaderWithAncestors($check, $req) => $e,
 			CheckedRequest::TransactionIndex($check, $req) => $e,
 			CheckedRequest::Receipts($check, $req) => $e,
 			CheckedRequest::Body($check, $req) => $e,
@@ -488,6 +525,15 @@ impl IncompleteRequest for CheckedRequest {
 					_ => Ok(()),
 				}
 			}
+			CheckedRequest::HeaderWithAncestors(ref check, ref req) => {
+				req.check_outputs(&mut f)?;
+
+				// make sure the output given is definitively a hash.
+				match check.block_hash {
+					Field::BackReference(r, idx) => f(r, idx, OutputKind::Hash),
+					_ => Ok(()),
+				}
+			}
 			CheckedRequest::TransactionIndex(_, ref req) => req.check_outputs(f),
 			CheckedRequest::Receipts(_, ref req) => req.check_outputs(f),
 			CheckedRequest::Body(_, ref req) => req.check_outputs(f),
@@ -508,15 +554,46 @@ impl IncompleteRequest for CheckedRequest {
 
 	fn complete(self) -> Result<Self::Complete, net_request::NoSuchOutput> {
 		match self {
-			CheckedRequest::HeaderProof(_, req) => req.complete().map(CompleteRequest::HeaderProof),
-			CheckedRequest::HeaderByHash(_, req) => req.complete().map(CompleteRequest::Headers),
-			CheckedRequest::TransactionIndex(_, req) => req.complete().map(CompleteRequest::TransactionIndex),
-			CheckedRequest::Receipts(_, req) => req.complete().map(CompleteRequest::Receipts),
-			CheckedRequest::Body(_, req) => req.complete().map(CompleteRequest::Body),
-			CheckedRequest::Account(_, req) => req.complete().map(CompleteRequest::Account),
-			CheckedRequest::Code(_, req) => req.complete().map(CompleteRequest::Code),
-			CheckedRequest::Execution(_, req) => req.complete().map(CompleteRequest::Execution),
-			CheckedRequest::Signal(_, req) => req.complete().map(CompleteRequest::Signal),
+			CheckedRequest::HeaderProof(_, req) => {
+				trace!(target: "on_demand", "HeaderProof request completed {:?}", req);
+				req.complete().map(CompleteRequest::HeaderProof)
+			}
+			CheckedRequest::HeaderByHash(_, req) => {
+				trace!(target: "on_demand", "HeaderByHash request completed {:?}", req);
+				req.complete().map(CompleteRequest::Headers)
+			}
+			CheckedRequest::HeaderWithAncestors(_, req) => {
+				trace!(target: "on_demand", "HeaderWithAncestors request completed {:?}", req);
+				req.complete().map(CompleteRequest::Headers)
+			}
+			CheckedRequest::TransactionIndex(_, req) => {
+				trace!(target: "on_demand", "TransactionIndex request completed {:?}", req);
+				req.complete().map(CompleteRequest::TransactionIndex)
+			}
+			CheckedRequest::Receipts(_, req) => {
+				trace!(target: "on_demand", "Receipt request completed {:?}", req);
+				req.complete().map(CompleteRequest::Receipts)
+			}
+			CheckedRequest::Body(_, req) => {
+				trace!(target: "on_demand", "Block request completed {:?}", req);
+				req.complete().map(CompleteRequest::Body)
+			}
+			CheckedRequest::Account(_, req) => {
+				trace!(target: "on_demand", "Account request completed {:?}", req);
+				req.complete().map(CompleteRequest::Account)
+			}
+			CheckedRequest::Code(_, req) => {
+				trace!(target: "on_demand", "Code request completed {:?}", req);
+				req.complete().map(CompleteRequest::Code)
+			}
+			CheckedRequest::Execution(_, req) => {
+				trace!(target: "on_demand", "Execution request completed {:?}", req);
+				req.complete().map(CompleteRequest::Execution)
+			}
+			CheckedRequest::Signal(_, req) => {
+				trace!(target: "on_demand", "Signal request completed {:?}", req);
+				req.complete().map(CompleteRequest::Signal)
+			}
 		}
 	}
 
@@ -552,6 +629,9 @@ impl net_request::CheckedRequest for CheckedRequest {
 			CheckedRequest::HeaderByHash(ref prover, _) =>
 				expect!((&NetResponse::Headers(ref res), &CompleteRequest::Headers(ref req)) =>
 					prover.check_response(cache, &req.start, &res.headers).map(Response::HeaderByHash)),
+			CheckedRequest::HeaderWithAncestors(ref prover, _) =>
+				expect!((&NetResponse::Headers(ref res), &CompleteRequest::Headers(ref req)) =>
+					prover.check_response(cache, &req.start, &res.headers).map(Response::HeaderWithAncestors)),
 			CheckedRequest::TransactionIndex(ref prover, _) =>
 				expect!((&NetResponse::TransactionIndex(ref res), _) =>
 					prover.check_response(cache, res).map(Response::TransactionIndex)),
@@ -585,6 +665,8 @@ pub enum Response {
 	HeaderProof((H256, U256)),
 	/// Response to a header-by-hash request.
 	HeaderByHash(encoded::Header),
+	/// Response to a header-by-hash with ancestors request.
+	HeaderWithAncestors(Vec<encoded::Header>),
 	/// Response to a transaction-index request.
 	TransactionIndex(net_request::TransactionIndexResponse),
 	/// Response to a receipts request.
@@ -626,6 +708,10 @@ pub enum Error {
 	Decoder(::rlp::DecoderError),
 	/// Empty response.
 	Empty,
+	/// Response data length exceeds request max.
+	TooManyResults(u64, u64),
+	/// Response data is incomplete.
+	TooFewResults(u64, u64),
 	/// Trie lookup error (result of bad proof)
 	Trie(TrieError),
 	/// Bad inclusion proof
@@ -642,6 +728,8 @@ pub enum Error {
 	WrongTrieRoot(H256, H256),
 	/// Wrong response kind.
 	WrongKind,
+	/// Wrong sequence of headers.
+	WrongHeaderSequence,
 }
 
 impl From<::rlp::DecoderError> for Error {
@@ -702,6 +790,65 @@ impl HeaderProof {
 	}
 }
 
+/// Request for a header by hash with a range of ancestors.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct HeaderWithAncestors {
+	/// Hash of the last block in the range to fetch.
+	pub block_hash: Field<H256>,
+	/// Number of headers before the last block to fetch in addition.
+	pub ancestor_count: u64,
+}
+
+impl HeaderWithAncestors {
+	/// Check a response for the headers.
+	pub fn check_response(
+		&self,
+		cache: &Mutex<::cache::Cache>,
+		start: &net_request::HashOrNumber,
+		headers: &[encoded::Header]
+	) -> Result<Vec<encoded::Header>, Error> {
+		let expected_hash = match (self.block_hash, start) {
+			(Field::Scalar(ref h), &net_request::HashOrNumber::Hash(ref h2)) => {
+				if h != h2 { return Err(Error::WrongHash(*h, *h2)) }
+				*h
+			}
+			(_, &net_request::HashOrNumber::Hash(h2)) => h2,
+			_ => return Err(Error::HeaderByNumber),
+		};
+
+		let start_header = headers.first().ok_or(Error::Empty)?;
+		let start_hash = start_header.hash();
+		if start_hash != expected_hash {
+			return Err(Error::WrongHash(expected_hash, start_hash));
+		}
+
+		let expected_len = 1 + cmp::min(self.ancestor_count, start_header.number());
+		let actual_len = headers.len() as u64;
+		match actual_len.cmp(&expected_len) {
+			cmp::Ordering::Less =>
+				return Err(Error::TooFewResults(expected_len, actual_len)),
+			cmp::Ordering::Greater =>
+				return Err(Error::TooManyResults(expected_len, actual_len)),
+			cmp::Ordering::Equal => (),
+		};
+
+		for (header, prev_header) in headers.iter().zip(headers[1..].iter()) {
+			if header.number() != prev_header.number() + 1 ||
+				header.parent_hash() != prev_header.hash()
+			{
+				return Err(Error::WrongHeaderSequence)
+			}
+		}
+
+		let mut cache = cache.lock();
+		for header in headers {
+			cache.insert_block_header(header.hash(), header.clone());
+		}
+
+		Ok(headers.to_vec())
+	}
+}
+
 /// Request for a header by hash.
 #[derive(Debug, Clone, PartialEq, Eq)]
 pub struct HeaderByHash(pub Field<H256>);
@@ -935,11 +1082,12 @@ mod tests {
 	use ethereum_types::{H256, Address};
 	use memorydb::MemoryDB;
 	use parking_lot::Mutex;
-	use trie::{Trie, TrieMut, SecTrieDB, SecTrieDBMut};
-	use trie::recorder::Recorder;
+	use trie::{Trie, TrieMut};
+	use ethtrie::{SecTrieDB, SecTrieDBMut};
+	use trie::Recorder;
 	use hash::keccak;
 
-	use ethcore::client::{BlockChainClient, BlockInfo, TestBlockChainClient, EachBlockWith};
+	use ::ethcore::client::{BlockChainClient, BlockInfo, TestBlockChainClient, EachBlockWith};
 	use ethcore::header::Header;
 	use ethcore::encoded;
 	use ethcore::receipt::{Receipt, TransactionOutcome};
@@ -993,6 +1141,83 @@ mod tests {
 		assert!(HeaderByHash(hash.into()).check_response(&cache, &hash.into(), &[raw_header]).is_ok())
 	}
 
+	#[test]
+	fn check_header_with_ancestors() {
+		let mut last_header_hash = H256::default();
+		let mut headers = (0..11).map(|num| {
+			let mut header = Header::new();
+			header.set_number(num);
+			header.set_parent_hash(last_header_hash);
+
+			last_header_hash = header.hash();
+			header
+		}).collect::<Vec<_>>();
+
+		headers.reverse();  // because responses are in reverse order
+
+		let raw_headers = headers.iter()
+			.map(|hdr| encoded::Header::new(::rlp::encode(hdr).into_vec()))
+			.collect::<Vec<_>>();
+
+		let mut invalid_successor = Header::new();
+		invalid_successor.set_number(11);
+		invalid_successor.set_parent_hash(headers[1].hash());
+
+		let raw_invalid_successor = encoded::Header::new(::rlp::encode(&invalid_successor).into_vec());
+
+		let cache = Mutex::new(make_cache());
+
+		let header_with_ancestors = |hash, count| {
+			HeaderWithAncestors {
+				block_hash: hash,
+				ancestor_count: count
+			}
+		};
+
+		// Correct responses
+		assert!(header_with_ancestors(headers[0].hash().into(), 0)
+				.check_response(&cache, &headers[0].hash().into(), &raw_headers[0..1]).is_ok());
+		assert!(header_with_ancestors(headers[0].hash().into(), 2)
+				.check_response(&cache, &headers[0].hash().into(), &raw_headers[0..3]).is_ok());
+		assert!(header_with_ancestors(headers[0].hash().into(), 10)
+				.check_response(&cache, &headers[0].hash().into(), &raw_headers[0..11]).is_ok());
+		assert!(header_with_ancestors(headers[2].hash().into(), 2)
+				.check_response(&cache, &headers[2].hash().into(), &raw_headers[2..5]).is_ok());
+		assert!(header_with_ancestors(headers[2].hash().into(), 10)
+				.check_response(&cache, &headers[2].hash().into(), &raw_headers[2..11]).is_ok());
+		assert!(header_with_ancestors(invalid_successor.hash().into(), 0)
+				.check_response(&cache, &invalid_successor.hash().into(), &[raw_invalid_successor.clone()]).is_ok());
+
+		// Incorrect responses
+		assert_eq!(header_with_ancestors(invalid_successor.hash().into(), 0)
+				   .check_response(&cache, &headers[0].hash().into(), &raw_headers[0..1]),
+				   Err(Error::WrongHash(invalid_successor.hash(), headers[0].hash())));
+		assert_eq!(header_with_ancestors(headers[0].hash().into(), 0)
+				   .check_response(&cache, &headers[0].hash().into(), &[]),
+				   Err(Error::Empty));
+		assert_eq!(header_with_ancestors(headers[0].hash().into(), 10)
+				   .check_response(&cache, &headers[0].hash().into(), &raw_headers[0..10]),
+				   Err(Error::TooFewResults(11, 10)));
+		assert_eq!(header_with_ancestors(headers[0].hash().into(), 9)
+				   .check_response(&cache, &headers[0].hash().into(), &raw_headers[0..11]),
+				   Err(Error::TooManyResults(10, 11)));
+
+		let response = &[raw_headers[0].clone(), raw_headers[2].clone()];
+		assert_eq!(header_with_ancestors(headers[0].hash().into(), 1)
+				   .check_response(&cache, &headers[0].hash().into(), response),
+				   Err(Error::WrongHeaderSequence));
+
+		let response = &[raw_invalid_successor.clone(), raw_headers[0].clone()];
+		assert_eq!(header_with_ancestors(invalid_successor.hash().into(), 1)
+				   .check_response(&cache, &invalid_successor.hash().into(), response),
+				   Err(Error::WrongHeaderSequence));
+
+		let response = &[raw_invalid_successor.clone(), raw_headers[1].clone()];
+		assert_eq!(header_with_ancestors(invalid_successor.hash().into(), 1)
+				   .check_response(&cache, &invalid_successor.hash().into(), response),
+				   Err(Error::WrongHeaderSequence));
+	}
+
 	#[test]
 	fn check_body() {
 		use rlp::RlpStream;