One way to detect bugs is via fuzzing and I initially requested it at #1650. I would like to revisit this and what should we focus next to build a secure Wasm runtime.
There are various successful fuzzer integration with Substrate components. However, I don’t think we have something that’s able to fuzz the whole Wasm runtime, instead of individual component.
My ideal fuzzing tool should be able to:
- Take a snapshot of a chain by downloading all the state
- Mutate the state to inject test account & other necessary changes
- Supply the state & wasm to the fuzzer
a. It will be great if we can use the production runtime as it is but it is ok if we need to provide a debug build
- Fuzzer generate valid block with help of the metadata
- Execute the block with a special test node. e.g. with mock signature verification host
- Execute some additional logic to validate the onchain state. e.g. assert total issuance delta matches to changed balances. This could be either part of the debug wasm runtime, or part of the fuzzer
- Gather coverage information and feedback to fuzzer
- Back to step 4
Step 1, 2, 3 are trivial.
Step 4 needs some work but not hard. I will image we want to build something similar to wasm-smith
Step 5 is basically implemented by Chopsticks.
Step 6 could be part of
try-runtime or just bunch additional runtime-specific code
Step 7 I am not sure if it is supported by wasmtime. However given we can do gas metering, there is no reason we cannot make it dump code coverage information.
And then we just need to find the right tool that is able to guide step 4 from result of step 6.
So my questions are:
- Are there any existing tool that’s able to generate code coverage for Wasm execution?
- If no, what will be the best starting point to make this tool exists?
- What will be the most suitable fuzzer we should try to integrate for this purpose? In the end, I want to fuzz the Wasm code, not the Wasm engine. I feel most of the existing tooling makes fuzzing the Wasm engine easy but not the actual Wasm code.