Developers have worked with Ethereum will know how useful it is to be able to fork the Eth mainnet and simulate transactions with it. Similarly, we want similar tool for Substrate networks.
We have various tools to simulate transaction, fork mainnet into testnet, but not yet a tool achieve a similar developer experience as Ethereum development tools.
Hereby I am presenting Chopsticks - The Substrate testing client to help you create a parallel reality of any Substrate network.
The goal of Chopsticks is to allow developers to fork any Substrate based networks instantly and be able to modify the onchain state of the fork net arbitrary as well as stub host functions. For example, replace council members, or stub the ext_crypto_sr25519_verify_version_2 host call to make it always return true.
Chopsticks is only one week old but it is already a MVP that’s able to fork any Substrate network that’s using Aura consensus algorithm (i.e. most of the parachains). (I only tested on Acala & Karura, but the codebase have zero reference to Acala so it should work with other parachains using AURA. Open an issue if that’s not the case.)
How it works
Chopsticks reimplements a subset of a Substrate node, and deliberately not implementing most of the verification mechanism so it can accept invalid blocks. For example, it does not check inherents so there is no need to sproof relaychain.
There are two components:
A nodejs server implements a subset of Substrate RPC and most of the logics
A rust Wasm executor that’s utilizing smoldot to execute Wasm calls and send result back to the nodejs server
The nodejs server implements a WebSocket Substrate JSONRPC client, implements a simple blockchain data structure and state overlay that is transparently forwarding requests to remote RPC node to fetch data. This means it only fetch required data to avoid pulling all the onchain state, which can be large.
The nodejs server will spawn the rust Wasm executor to execute runtime calls. The executor uses smoldot to execute the wasm runtime call and callback into the nodejs server when it is trying to access state. The state is either coming from remote RPC node, or the local blockchain representation. The nodejs server is able to intercept and modify the data if needed.
Next Steps
Implements more Substrate RPC endpoints
Allow stub host calls to disable signature verification
Implements API to modify onchain storage
Supports BABE consensus so this can be used to produce Polkadot/Kusama blocks
Fork relaychain and multiple parachains and implements XCM channel to allow test XCM related code
Compile the rust executor into Wasm to avoid cross process communication and simplify deployment
I think it is a great idea and I think including support to disable signature verification (we talked about this in our Summit with @shawntabrizi ) is going to allow a lot of test/dev uses cases.
However I’m afraid this is going to be another fork system in place (we also have our own) that won’t be able to be maintained. There are already many forking solutions but as far as I remember, each one is having some issues and is not really maintained.
If we want something where all the community can easily contribute, maybe it would be better to think of something embedded in substrate and in the nodes. Things like try-runtime or follow-chain could definitely allow some use cases, and if some features like “disabling signature verification” and being compatible with light client, it would become very useful.
Maybe we could even come with a trait that authoring pallet could implement allowing to bypass part of the authoring (but this would have to be very carefully designed )
I am not aware any fork system that meets my requirements so if you have something to share, please do.
I have done my research and PoCs and it is simply impossible to build this kind of tool within Substrate at this stage.
One of the goal is that I want to build this without pulling in any specific runtime, otherwise it will be hard to make it runtime agnostic. And it is just not possible to build a client with Substrate without native runtime. And fix it requires multiple significant refactoring.
Another reason a nodejs server is required is that, again, I want this to be runtime agnostic. For some tasks such as building new blocks, I need the ability to encode/decode extrinsic, which requires the ability to parse metadata type info and construct the data accordingly. I am not aware any Rust tooling for that and it is just much easier use polkadot.js for it. In the end, it makes perfect sense to use a dynamic typing language to handling dynamic types.
I also reuse smoldot for the wasm execution, so there isn’t much duplicated work regards on the runtime execution part. In fact, I contributed some PR to smoldot while using it to build this.
I think end of the day, different tools, similar to what Bryan has put together here are useful and will complement each other. As of now, I don’t see major overlap between this and try-runtime either. If we come to realize that they are literally doing the same thing though, perhaps we should gather forces. I don’t see that being the case for now.
What I like to vouch here more than try-runtime CLI is the little bit of logic that lives in substrate and is feature gated try-runtime. These are standards that we are trying to push forward and are 10x more important a particular CLI that interacts with them. I am still a big fan of all the bits and pieces that feature = try-runtime adds to substrate. To remind you of a few:
pre-upgrade and post-upgrade.
try_state hook.
All of which are executable through the trait TryRuntime runtime API, which I hope to see more and more actually utilize. I hope to soon add some fuzzing functionality to runtime as well under the same feature flag.
These are the real fundamental abstractions that we should make sure we are not repeating. Also, these, as Alan said, indeed better live inside of substrate to remain a standard. I would love to see actually more tools using these APIs, rather than narrowing it down to try-runtime-cli being the only tool.
The idea behind all the feature = try-runtime code is to enable a runtime to have the same behavior, but be more much debuggable. For example, TryRuntime runtime API also has an execute-block function, but it allows the call site to do more things. I would like to see more ideas added to this trait actually.
Thank you @kianenigma , that is what I meant by using substrate and the try-runtime. I already see how we could allow additional features like non-verified extrinsics into it or analyzers like callgrind
The only drawback that I see (and the reason why we maintain our fork tests) is that it runs a custom client and runtime, leading to differences with real use cases (like the state_version issue we discussed recently)
I don’t know yet your usage of polkadot.js but if you want to stay Rusty, it sounds like our SCALES library(a bit beta still) might help with the (de)serialization of data dynamically. I created that a while back to make our client Sube runtime agnostic and not depend on code generation like Subxt does.
What’s your position on Zombienet vs parachain-launch?
I was considering Zombienet would be the new standard and was pushing to get the state manipulator compatible with it
Just for adding more context about ZombieNet, we are planning to start working in the new version with a draft roadmap and one of the features we want to add is allow to spawn (and test) from a live network, we also want to add more flexibility in the parachains configuration to cover more use cases and make the tool more easy to integrate for the teams.
This is a tool we use internally and we plan to continue develop and maintain, so we are happy to explore the needs from other teams and add the needed features.
Zombienet is very heave weight compare to parachain-launch so I feel they fits in different category.
For more advanced monitoring etc, zombienet will be the solution but for quick 10 min testnet to verify an upgrade wasm file, parachain-launch works perfectly fine.
I think Zombienet will cover all the use cases of parachian-launch but I don’t think we want to deprecate parachian-launch until Zombienet is fully mature & stable.
Thanks @xlc, I’m currently researching the working options to see if we can integrate some to ZombieNet or take inspiration of them. If you have a working PoC to check would be great.
)
