How to make a fuzz er for Wasm runtime

Tech Talk
1

One way to detect bugs is via fuzzing and I initially requested it at #1650. I would like to revisit this and what should we focus next to build a secure Wasm runtime.

There are various successful fuzzer integration with Substrate components. However, I don’t think we have something that’s able to fuzz the whole Wasm runtime, instead of individual component.

My ideal fuzzing tool should be able to:

  1. Take a snapshot of a chain by downloading all the state
  2. Mutate the state to inject test account & other necessary changes
  3. Supply the state & wasm to the fuzzer
    a. It will be great if we can use the production runtime as it is but it is ok if we need to provide a debug build
  4. Fuzzer generate valid block with help of the metadata
  5. Execute the block with a special test node. e.g. with mock signature verification host
  6. Execute some additional logic to validate the onchain state. e.g. assert total issuance delta matches to changed balances. This could be either part of the debug wasm runtime, or part of the fuzzer
  7. Gather coverage information and feedback to fuzzer
  8. Back to step 4

Step 1, 2, 3 are trivial.
Step 4 needs some work but not hard. I will image we want to build something similar to wasm-smith
Step 5 is basically implemented by Chopsticks.
Step 6 could be part of try-runtime or just bunch additional runtime-specific code
Step 7 I am not sure if it is supported by wasmtime. However given we can do gas metering, there is no reason we cannot make it dump code coverage information.
And then we just need to find the right tool that is able to guide step 4 from result of step 6.

So my questions are:

  • Are there any existing tool that’s able to generate code coverage for Wasm execution?
  • If no, what will be the best starting point to make this tool exists?
  • What will be the most suitable fuzzer we should try to integrate for this purpose? In the end, I want to fuzz the Wasm code, not the Wasm engine. I feel most of the existing tooling makes fuzzing the Wasm engine easy but not the actual Wasm code.
2

I have a massive list of items to implement on try-runtime in Try-runtime Revamp and Facelift by kianenigma · Pull Request #12537 · paritytech/substrate · GitHub so it might be a few weeks I can finish it, but as a follow-up I can foresee adding fuzzing there as well. From a technical PoV, try-runtime is rather suitable for this, since it is pretty much a WasmExecutor.

I don’t know a lot about how to achieve good coverage yet. I am experimenting with turning the fuzzing tests of pallet-nomination-pools into honggfuzz, and use the arbitrary crate. So far, I have a bit of custom code that generates appropriate pallet Calls from a random seed and have not experimented if/how I can achieve a good coverage through feedback. It would be great to figure this step out, because otherwise we would need a custom “generate random call based on seed” for every pallet’s call, form which we can generate a “random block”.

1 Like