Forklift: a tool to test reorgs locally

Tech Talk
1

When building PAPI, we’ve often found it challenging to write tests for use cases that need to handle reorgs. Reorgs happen when there are competing forks before one of them is finalized. In those cases, state can diverge between branches, so clients need to handle that correctly.

Today, testing against Polkadot chains mostly comes down to two great tools: Zombienet and Chopsticks. Both are well-suited for workflows like:

fork a chain, modify some state, produce blocks, inspect results.

But neither is a great fit for tests that need multiple live branches:

  • Zombienet spins up a local chain with multiple block producers based on a Polkadot SDK node. Competing forks and reorgs can happen because it simulates a real network, but there’s no reliable way to force a particular state or reorg.
  • Chopsticks creates a local copy of a chain and lets you produce blocks on demand. However, it is centered around a single head that gets finalized immediately.

With the recent development of PAPI Rescue, which takes competing forks into account when defending against an attacker, we ran into this limitation again. So in my spare time I built Forklift, a new chain-forking tool that supports multiple branches.

Forklift is inspired by AcalaNetwork’s Chopsticks, but it is built with PAPI. It also uses @acala-network/chopsticks-executor, the package that executes runtime functions locally on top of Smoldot.

What Forklift does

  • Forks a live chain from a remote WebSocket endpoint
  • Supports multiple concurrent branches, including competing forks and reorgs
  • Uses immutable Merkle-trie-backed storage with structural sharing between blocks
  • Includes helpers for relay / parachain wiring for local XCM testing, even after the chain has already started.
  • Uses the newer chainHead_v1 / archive_v1 RPC methods rather than legacy RPCs
  • Provides a YAML-based CLI config for single-chain and multi-chain setups

Quick example

Install it with:

pnpm i @polkadot-api/forklift

Run it directly from a live endpoint:

pnpm forklift wss://polkadot-asset-hub-rpc.polkadot.io

For multi-chain / XCM setups, the intended interface is a YAML config file. For example:

chains:
  relay:
    endpoint: wss://rpc.polkadot.io
    port: 3000

  assetHub:
    endpoint: wss://polkadot-asset-hub-rpc.polkadot.io
    port: 3001
    parachainOf: relay

  bridgeHub:
    endpoint: wss://sys.ibp.network/bridge-hub-polkadot
    port: 3002
    parachainOf: relay

And then run:

pnpm forklift -c config.yaml

Creating forks

Forklift tries to stay as compatible as possible with Chopsticks’ API. By default, blocks are produced shortly after a new transaction is submitted, but instead of being finalized immediately, they are finalized after 2 seconds. These settings can be customized through the YAML config or through the programmatic API.

You can also create blocks explicitly through the dev_newBlock RPC. This RPC accepts an extra parameter, type: 'best' | 'finalized' | 'fork', which controls how that block is treated.

As an example, we can create competing forks like this:

import { Binary } from "polkadot-api";
import { createWsClient } from "polkadot-api/ws";

const client = createWsClient("ws://localhost:3000");

const prettyHash = (hash: string) => `${hash.substring(0, 6)}...`;

// Wait for the initial finalized block
const initialFinalized = await client.getFinalizedBlock();

// Subscribe to best blocks so we can observe branch changes
client.bestBlocks$.subscribe((bestBranch) => {
  console.log(
    "Best branch is now",
    bestBranch.map((b) => prettyHash(b.hash)).reverse()
  );
});

// Assume `alice` is a signer available in your test setup.
// Add funds to ALICE
await client._request("dev_setStorage", [
  [
    [
      "0x26aa394eea5630e07c48ae0c9558cef7b99d880ec681799c0cf30e8886371da9de1e86a9a8c739864cf3cc5ec2bea59fd43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d",
      "0x151900003f0000000100000001000000401edad7e84100000000000000000000b03596d71721000000000000000000000000000000000000000000000000000000000000000000000000000000000080",
    ],
  ],
]);

// We will keep track of two fork heads and extend them independently
const forks = [initialFinalized.hash, initialFinalized.hash];

for (let i = 0; i < 3; i++) {
  for (let f = 0; f < forks.length; f++) {
    const parent = forks[f];

    // Use a different transaction on each branch so the resulting blocks differ
    const tx = await client
      .getUnsafeApi()
      .tx.System.remark({
        remark: Binary.fromText(`Iteration ${i}-${f}`),
      })
      .sign(alice);

    const newHash = await client._request("dev_newBlock", [
      {
        type: "best",
        parent,
        transactions: [Binary.toHex(tx)],
      },
    ]);

    console.log(`new block for fork ${f}: ${prettyHash(newHash)}`);

    // Update that fork head
    forks[f] = newHash;
  }
}

client.destroy();

There’s more

Forklift is still in its early stages, but it already has a few more features: XCM messaging across chains, storage diffs between arbitrary blocks thanks to the shared Merkle-trie storage structure, the ability to replace RPC methods through the programmatic interface, and more.

It is still missing many features from other tools, especially Chopsticks. The ones I miss the most are persisting state in a database and bootstrapping from genesis. We’ll get there. For now, though, it already covers what we need for developing PAPI.

I hope it can be useful for other teams too.

2

Kudos on the name :raising_hands: