Subxt v0.50.0: Adding Historic Block Support & More!

Tech Talk
1

Hey everyone!

Long short short, a huge Subxt v0.50.0 PR is ready to try out, and I’ve started publishing beta versions to crates.io so that you can get your hands on this huge upcoming release and start providing feedback! Check out the CHANGELOG for details!

The version bump here is from 0.44 to 0.50 to signify the extent of the release and the fact that it is a more significant step to upgrade to than previous releases.

Short story long, I’d like to highlight some of the major changes since previous versions and why they have happened (but please consult the CHANGELOG for more)!

Support for historic blocks

TL;DR this is what working with historic blocks looks like now using Subxt v0.50.0:

use subxt::{Error, OnlineClient, PolkadotConfig};

#[tokio::main]
async fn main() -> Result<(), Error> {
    // To access old blocks we point to an archive RPC node:
    let api = OnlineClient::<PolkadotConfig>::from_url("wss://rpc.polkadot.io").await?;

    // Pick some arbitrary old blocks to work with
    for block_number in 1234567u32.. {
        // This instantiates an API which is working at a specific block.
        let at_block = api.at_block(block_number).await?;

        // Now we can ask for extrinsics, storage etc at this block..
        let extrinsics = at_block.extrinsics().fetch().await?;
        let storage_value = at_block.storage().entry(some_entry).fetch((key1, key2)).await?;
        let constant = at_block.constants().entry(some_entry)?;
    }

    Ok(())
}

Prior to this release, Subxt has always been “head-of-chain” only, meaning that it can work with current blocks and state, but has not supported older blocks. Users who have wanted to work with slightly older blocks have employed semi-supported hacks like fetching older metadata and using OnlineClient::set_metadata(older_metadata) to make Subxt able to understand things at the corresponding older blocks. Nevertheless, this was ugly, breaks certain APIs, and it would certainly not support anything older than V14 metadata.

At the same time, the only other way to work with historic blocks has been the Polkadot.js library and tooling built on top of this. Aside from questions over the ongoing maintenance of that library, having support for historic blocks in Rust as opposed to Javascript makes it possible to use it in many more platforms (via C bindings, for instance).

Historic blocks prior to V14 metadata need additional type information to be provided. We provide this type information via the lower level frame-decode library for some chains. We’ve put a lot of effort into porting type information from Polkadot.js (and even fixing it in a couple of places), but it will be an ongoing process to thoroughly test this information and gain confidence in it. For chains we haven’t already added information for, you’ll need to bring your own type information (we have used this tool to help find and debug missing types to build it up ourselves).

Configuration

In previous versions of Subxt, configuration existed at the type level only to provide type information about some chain, and was used like so:

use subxt::{OnlineClient, PolkadotConfig};

let api = OnlineClient::<PolkadotConfig>::new().await?;

Now, the configuration for Subxt has been extended to support working with historic blocks and so exists at the value level too. The above still works, but we can also construct and configure our configuration if we need to:

use subxt::{OnlineClient, PolkadotConfig};

let config = PolkadotConfig::builder()
    .use_historic_types(false)
    .build();
let api = OnlineClient::<PolkadotConfig>::new_with_config(config).await?;

The new Configuration is responsible for providing historic type information where needed, as well as spec version and transaction version information, and the metadata for a given spec version (this also means that configuration can opt to cache this information as it wishes).

As before, we provide PolkadotConfig and SubstrateConfig. The former works specifically on the Polkadot Relay Chain, and the latter works more broadly across a range of Substrate chains (but provides no specific type information for pre-V14 metadatas by default).

Constructing Transactions

Constructing and submitting transactions is kept very similar to older versions of Subxt. The only difference now is that we always select a block to work at first, and then construct transactions in the context of this block. For previous versions of Subxt, that block was always the latest one, but now the choice of block is more explicit.

Before we had:

let api = OnlineClient::<PolkadotConfig>::new().await?;

// Submit an extrinsic, waiting for success.
let events = api
    .tx()
    .sign_and_submit_then_watch_default(&balance_transfer_tx, &from)
    .await?
    .wait_for_finalized_success()
    .await?;

The block that we are working at is implicit here. In v0.50.0, we can still just “build a transaction in the context of the latest finalized block”:

let api = OnlineClient::<PolkadotConfig>::new().await?;

let events = api
    .tx()
    .await? // <- This is new; here we are fetching the latest block information to work at.
    .sign_and_submit_then_watch_default(&balance_transfer_tx, &from)
    .await?
    .wait_for_finalized_success()
    .await?;

We can also pick a specific block (which may be useful if you want to be explicit about how long a block can live for, for instance, or if you wish to build a transaction off a non-finalized block and take into account the current account nonce etc at that block):

let api = OnlineClient::<PolkadotConfig>::new().await?;

// Work at a specific block:
let at_block = api.at_block(12345).await?;

// This bit now looks the same as before:
let events = at_block
    .tx()
    .sign_and_submit_then_watch_default(&balance_transfer_tx, &from)
    .await?
    .wait_for_finalized_success()
    .await?;

Working with Storage Entries

Prior to v0.50.0, working with storage entries leant heavily on the generated code, and was quite restrictive in how values could be fetched or iterated over. Subxt v0.50.0 adds much more functionality around accessing storage entries, iterating and fetching from them at different levels, and decoding keys and values in flexible ways. These sorts of APIs are especially important as we extend to supporting historic blocks, and want as much power as possible without any statically generated code.

Before v0.50.0, working with storage entries looked like this:

let api = OnlineClient::<PolkadotConfig>::new().await?;

//// Fetching:
let result = api
    .storage()
    .at_latest()
    .await?
    .fetch(&storage_query)
    .await?;

//// Iterating
let mut results = api
    .storage()
    .at_latest()
    .await?
    .iter(storage_query)
    .await?;
while let Some(Ok(kv)) = results.next().await {
    println!("Keys decoded: {:?}", kv.keys); // <- Broken in some cases
    println!("Key: 0x{}", hex::encode(&kv.key_bytes));
    println!("Value: {:?}", kv.value);
}

Now, the APis look like this:

let api = OnlineClient::<PolkadotConfig>::new().await?;
let at_block = api.at_current_block().await?;

// Access an entry (but not any specific values within it):
let account_balances = at_block
    .storage()
    .entry(storage_query)?;

//// Fetching:

// We can fetch multiple values from an entry:
let value1 = account_balances.fetch((account_id1,)).await?;
let value2 = account_balances.fetch((account_id2,)).await?;

// Entries can be decoded into the static type given by the address:
let result = value1.decode()?;
// Or they can be decoded into any arbitrary shape:
let result = value1.decode_as::<scale_value::Value>()?;
// Or we can "visit" the entry for more control over decoding:
let result = value1.visit(my_visitor)?;
// Or we can just get the bytes out and do what we want:
let result_bytes = value1.bytes();


//// Iterating

// We can iterate over the same entry we fetched things from:
let mut balances = account_balances.iter(()).await?;
while let Some(Ok(entry)) = all_balances.next().await {
    let key = entry.key()?;
    let value = entry.value();

    // Decode the keys that can be decoded:
    let keys_tuple = key.decode()?;
    // Value is as above:
    let value = value.decode()?;

    println!("Keys decoded: {:?}", keys_tuple);
    println!("Key: 0x{}", hex::encode(key.bytes()));
    println!("Value: {:?}", value);
}

Working with dynamic inputs (ie, no codegen)

For writing CLI tools which access things depending on user input, it’s important to be able to use “dynamic” values rather than the statically generated APIs given by the #[subxt] macro. Subxt v0.50.0 provides more power in this area, which is important again for working at historic blocks that may not benefit from the #[subxt] macro.

Prior to v0.50.0, dynamic values relied heavily on the scale_value::Value type, like so:

let constant_query = subxt::dynamic::constant(
    "System", 
    "BlockLength"
);

let runtime_api_payload = subxt::dynamic::runtime_api_call(
    "AccountNonceApi",
    "account_nonce",
    vec![Value::from_bytes(account)],
);

let storage_query = subxt::dynamic::storage(
    "System", 
    "Account", 
    vec![Value::from_bytes(account)]
);

Now, owing to some internal improvements, these methods accept generic parameters to define the input and output types, which can make it much easier to encode and decode various things:

let constant_query = subxt::dynamic::constant::<Value>(
    "System", 
    "BlockLength"
);

let runtime_api_payload = subxt::dynamic::runtime_api_call::<_, Value>(
    "AccountNonceApi",
    "account_nonce",
    vec![Value::from_bytes(account)],
);
// We can provide more generic input args now, negating the need 
// to convert to Values unnecessarily:
let runtime_api_payload = subxt::dynamic::runtime_api_call::<_, Value>(
    "AccountNonceApi",
    "account_nonce",
    (account,),
);

// We no longer provide the keys up front for storage; we just point
// to the _entry_ we want and provide the key and return types:
let storage_query = subxt::dynamic::storage::<Vec<Value>, Value>(
    "System", 
    "Account",
);
// This allows us to set better key/value types if we know what to expect. Here
// we know what information we want from account info and the key format:
#[derive(scale_decode::DecodeAsType)]
struct MyAccountInfo {
    nonce: u32,
    data: MyAccountInfoData
}
#[derive(scale_decode::DecodeAsType)]
struct MyAccountInfoData {
    free: u128,
    reserved: u128
}
let storage_query = subxt::dynamic::storage::<(AccountId32,), MyAccountInfo>(
    "System", 
    "Account",
);

Metadata

Subxt works using a subxt_metadata::Metadata type. Previously, it was possible to convert V14 metadata and up into this type. Now, in order to support historic blocks, it’s possible to convert all of the available historic metadata versions into this type using these new APIs:

Metadata::from_v16(..)
Metadata::from_v15(..)
Metadata::from_v14(..)
Metadata::from_v13(..)
Metadata::from_v12(..)
Metadata::from_v11(..)
Metadata::from_v10(..)
Metadata::from_v9(..)
Metadata::from_v8(..)

Older versions require type information, and where type information is missing, we currently use a placeholder type that cannot be constructed in order to signal that we are missing the information in some areas.

No more subxt-core

In previous versions of Subxt, the subxt-core crate exported a no-std compatible subset of Subxt functionality. this was not required for WASM compatibility (Subxt supported and continues to support compiling to WASM) or generating C bindings, but could be useful for using Subxt in certain contexts which lack the std-provided operating system abilities.

Now, Subxt instead leans heavily on frame-decode to provide all of the lower level FRAME-specific encoding and decoding logic. frame-decode is independent from Subxt and is fully no-std by default, and so it is the library I would encourage people look into if they want certain Subxt functionality in a no-std context.

Making this change was in part done to make it much quicker to iterate on and build up this new version of Subxt. Previously, Subxt logic had been split across subxt and subxt-core crates such that parts of it could be provided in a no-std context, and this added a bunch of complexity to the codebase. Now, subxt is simpler to maintain and build on, and frame-decode provides at a lower level most of the encode and decode logic that Subxt relies on.

Next Steps

Currently, we have a beta version of this out in order to gather feedback and start putting it to use elsewhere.

In the near term, the remaining steps towards a full release are:

  • Improving the documentation and examples: given that this is such a large change, work is underway to re-do a lot of the documentation and examples. Some already exists, but there is much more to do.
  • Gathering and responding to feedback (please raise issues in the Subxt repository if you have any) which may result in fixes and new beta releases, or may be handled after the 0.50.0 release).

I am hoping to be able to do a full v0.50.0 release of Subxt at the end of January, but this will also depend on the feedback received, and may be delayed if anything significant is raised :slight_smile:

In the longer term, I’d like us to:

  • Work with the PAPI team to create a single metadata target that we can translate all metadata versions into. We do this internally in Subxt at the moment, but moving it to a separate repository would allow the PAPI team to also leverage this metadata translation and begin supporting older blocks.
  • Port extrinsic encoding into frame-decode, which would “complete” that crate in terms of it having all of the relevant encoding and decoding logic which Subxt relies on with full no-std compatibility.

Thank you for reading, and I hope you enjoy the new version of Subxt!

12 Likes