Test frameworks and utilities for Polkadot and friends

[s3krit]

As our ecosystem grows and many teams have needs for testing their work that often overlap (at least partially), I am interested in compiling a list of tools, utilities and frameworks that either already exist, or are actively being developed that can be used for performing testing and QA of runtimes, clients, and everything else!

If you are working on, or know of, any tools and projects not mentioned below please comment with a link to the tool and ideally a short description, so I can add them to the list. It can be focusing on a single project, or a more generic tool. Anything goes!

• Zombienet
  • GitHub - paritytech/zombienet: A cli tool to easily spawn ephemeral Polkadot/Substrate networks and perform tests against them.
  • Zombienet aims to be a testing framework for Substrate based blockchains, providing a simple cli tool that allows users to spawn and test ephemeral networks. Tests can be written in a DSL or JS (with polkadot.js)
• try-runtime
  • https://docs.substrate.io/reference/command-line-tools/try-runtime/
  • A tool that allows you to write tests against snapshots of runtime state
• Trappist
  • GitHub - paritytech/trappist
  • A set of tools/playground focusing on testing cross-chain functionality
• Chopsticks
  • GitHub - AcalaNetwork/chopsticks: Create parallel reality of your Substrate network.
  • Create a parallel reality of a live chain, and optionally set storage of your new chain for testing
• Acala’s e2e tests
  • GitHub - AcalaNetwork/e2e-tests
  • Great example of some really nicely-written e2e tests testing a variety of cross-chain functionality
• parachain-launch
  • GitHub - open-web3-stack/parachain-launch: Tool to generate docker-compose.yml to launch a full relaychain - parachain testnet.

[NachoPal]

• Parachains Integration Tests
  • GitHub - paritytech/parachains-integration-tests: Tool to make Parachains integrations tests development easier and faster
  • @parity/parachains-integration-tests - npm
  • For writing e2e test in yaml format, abstracting Mocha, Chia and Polkadot JS API implementations. It can connect to Zombienet or to any other testnet of your choice.

[xlc]

For XCM tests

• XCM simulator
  • Test XCM code with mock runtime
  • polkadot/xcm/xcm-simulator at master · paritytech/polkadot · GitHub
• XCM emulator
  • Test XCM code with real runtime
  • xcm-simulator/xcm-emulator at master · shaunxw/xcm-simulator · GitHub
  • Also WIP PR to merge this into Cumulus

[tomaka]

As I’ve mentioned in my latest smoldot treasury proposal, feel free to open issues/discussions in the smoldot repository if you need some specialized tools.

Examples that I have in mind: a tool that prints the runtime spec of a runtime blob, a DHT crawler (that enumerates all nodes on the network), a tool that prints the list of all validators that register themselves with authority-discovery, a tool that prints the PeerId and version of a node given an IP+port, a tool that prints the IP+port of a PeerId, etc.
Some of these examples would require a lot of efforts though, but they’re all theoretically feasible.

Given that smoldot has been designed to run in the browser, there’s the possibility to create a web page that proposes these tools, which is generally a better UX compared to having to compile some Rust source code.

However, since it’s efforts, I’d prefer to work on this only if there’s an actual demand, and not create these tools only for them to not be used.