Decoding old and new blocks and storage entries in Rust: October 2024 update

Tech Talk
1

Back in August I posted an update on the progress towards decoding historic blocks and storage entries in Rust.

The work planned for this half is now complete, and so I wanted to share with you the status, and pointers to the libraries and resources needed to enable you to decode old or new extrinsics and storage keys/values in Rust.

Let’s get into it!

The short version is:

  • Back in August, we were successfully decoding all historic relay chain blocks, but hadn’t started looking at storage decoding yet. Since then, we added storage decoding support to polkadot-historic-decoding-exaomple, and consequently expanded our type definitions a lot to be able to decode all of the new types we encountered.
  • The other goal we stated back in our August update was to then build a higher level library from these learnings. Now, we’ve built frame-decode in order to take the logic needed for decoding things and put it in one place for others to benefit from.

And now for the longer version:

frame-decode

A new frame-decode library has been released in the last few days. This library provides the high level functions capable of decoding extrinsics (V4 and V5) and storage keys/values. This is the culmination of the historic decoding work, and uses scale-info-legacy, scale-decode and other libraries under the hood, exposing a fairly simple interface combining everything.

To decode historic extrinsics or storage keys/values, you’ll need to provide some type information, which for the Polkadot relay chain is here. These types are capable of decoding all relay chain blocks and has been tested to decode every storage key and value at over 750 historic blocks spread across the different runtimes so far.

Here’s an example of decoding an extrinsic using historic type information (which assumes that you already have some metadata and extrinsic bytes to hand):

use frame_decode::extrinsics::decode_extrinsic_legacy;
use frame_metadata::RuntimeMetadata;
use parity_scale_codec::Decode;
use scale_info_legacy::ChainTypeRegistry;

let extrinsic_bytes = todo!("Obtain extrinsic bytes from node");
let metadata_bytes = todo!("Obtain metadata bytes from node");

// Decode our metadata. In this example we assume it's V12 metadata, but any version requiring
// historic types (V13 and below) would be handled the same.
let RuntimeMetadata::V12(metadata) = RuntimeMetadata::decode(&mut &*metadata_bytes).unwrap() else { panic!() };

// For historic types, we need to provide type definitions, since they aren't in the
// metadata. We use scale-info-legacy to do this, and have already defined types for the
// Polkadot relay chain, so let's load those in:
let historic_type_bytes = std::fs::read("frame-decode/types/polkadot_types.yaml").unwrap();
let historic_types: ChainTypeRegistry = serde_yaml::from_slice(&historic_type_bytes).unwrap();

// We configure the loaded types for the spec version of the extrinsics we want to decode,
// because types can vary between different spec versions. Here we assume that our extrinsic
// comes from a block with spec_version of 30.
let mut historic_types_for_spec = historic_types.for_spec_version(30);

// We also want to add to these types any relevant information from the metadata itself. This avoids
// needing to hardcode a load of type definitions that we can otherwise infer from metadata.
let types_from_metadata = frame_decode::helpers::type_registry_from_metadata(&metadata).unwrap();
historic_types_for_spec.prepend(types_from_metadata);

// With this, we can decode the extrinsic bytes into information about the parts of the extrinsic:
let ext_info = decode_extrinsic_legacy(&mut &*extrinsic_bytes, &metadata, &historic_types_for_spec).unwrap();

// Now, armed with this information, we can access the bytes for each part of the extrinsic,
// as well as any associated type information, and decode or view them however we like:

// Decode the signature details to scale_value::Value's.
if let Some(sig) = ext_info.signature_payload() {
    let address_bytes =  &ext_bytes[sig.address_range()];
    let address_value = decode_with_visitor(
        &mut &*address_bytes,
        *sig.address_type(),
        &metadata.types,
        ValueVisitor::new()
    ).unwrap();

    let signature_bytes = &ext_bytes[sig.signature_range()];
    let signature_value = decode_with_visitor(
        &mut &*signature_bytes,
        *sig.signature_type(),
        &metadata.types,
        ValueVisitor::new()
    ).unwrap();
}

// Decode the transaction extensions to scale_value::Value's.
if let Some(exts) = ext_info.transaction_extension_payload() {
    for ext in exts.iter() {
        let ext_name = ext.name();
        let ext_bytes = &ext_bytes[ext.range()];
        let ext_value = decode_with_visitor(
            &mut &*ext_bytes,
            *ext.ty(),
            &metadata.types,
            ValueVisitor::new()
        ).unwrap();
    }
}

// Decode the call data args to scale_value::Value's.
for arg in ext_info.call_data() {
    let arg_name = arg.name();
    let arg_bytes = &ext_bytes[arg.range()];
    let arg_value = decode_with_visitor(
        &mut &*arg_bytes,
        *arg.ty(),
        &metadata.types,
        ValueVisitor::new()
    ).unwrap();
}

To decode modern (ie runtimes with >=V14 metadata) extrinsics and storage, you only need the runtime metadata, which can be obtained from the chain via the runtime API Metadata_metadata_at_version(version).

frame-decode was recently integrated into Subxt so that this extrinsic decoding logic is not duplicated in two places (which is helpful now that V5 extrinsics are going to be a thing), and Subxt need not care about the lower level details.

frame-decode has also been integrated into polkadot-historic-decoding-exaomple, which has allowed us to verify that it works for decoding blocks and storage entries (and generally helps to keep that crate simpler).

polkadot-historic-decoding-example

polkadot-historic-decoding-exaomple is an example binary that basically exists to check that we can decode and print information about blocks and storage entries on a node (primarily it’s geared towards talking to Polkadot relay chain nodes, but can be pointed elsewhere).

This code makes use of frame-decode, and is probably the best example of the end to end logic needed to talk to a chain and decode data from it. Where decoding of some type fails, it’s also a good tool too use to see a detailed error message about what went wrong and what the values we could decode look like. There is also a js folder in the repository which has a small PJS CLI tool which can be used to compare output with the Rust decoding.

The code is more complex than needed in order to eg parallelize requests for blocks/storage entries and provide good error feedback, but is a good starting point nonetheless! We’ll aim to add some much simpler end to end examples to the frame-decode repository so that it’s clearer how to get started!

How to contribute

frame-decode is a fairly new library, and so feedback in the form of issues in [the github repository]((GitHub - paritytech/frame-decode: Deocde extrinsics and storage entries from runtimes using frame-metadata) is very welcome. The library was built to be useful in both the historic decoding example and in Subxt, which has helped a bunch with building the right APIs, but we’re keen to support as many use cases as possible and might have missed something! Equally, any bug reports are very welcomed!

For decoding historic data, we maintain the type information for the Polkadot relay chain. For now, we don’t have the bandwidth to build up type information for other chains (though eventually we’d like to support Kusama too). This means that:

  • If you run into any issues decoding runtime storage entries or blocks on the relay chain using our type information, please open issues in frame-decode so that we can get them sorted out for you!
  • If you’d like to decode types on other chains, feel free to get in touch and we can help walk you through how to go about building up the relevant type information (there’s a little info here in the August update). It would be amazing to see type information built up for other chains, and one of the nice things is that once the type information is there, it will only ever need updating in the event of an issue or missing type.
8 Likes