Over the last few months, we in the Subxt team (well, mainly Alex :)) have introduced a V15 metadata format to Substrate and a couple of new Runtime API calls for accessing it. This V15 metadata format is currently unstable and subject to change. The aims of this post are to:
- Summarize some of the key changes made as part of introducing this new version.
- Mention the upcoming changes we’ll be making next.
- Propose a date and approach for stabilizing V15 metadata.
For those that don’t already know, the metadata obtained from a node is essentially a SCALE encoded blob that decodes to a
frame_metadata::RuntimeMetadataPrefixed object (see frame-metadata/lib.rs at 438a5b098bb9d5b5a09bdc5b68275b2c5e63a010 · paritytech/frame-metadata · GitHub to see the exact type it decodes into).
The primary purpose of metadata (as I see it) is to provide all of the information necessary to be able to interact with a generic Substrate based node via its RPC APIs. It currently includes:
- The information necessary to construct valid transactions that can be submitted via calls like
- The information necessary to decode errors and events that are returned from the node during extrinsic submission.
- The information necessary to build valid storage keys and decode the storage items living at those locations.
- The information necessary to access and decode any constant values declared in pallets.
Any data that is expected to be the same across all Substrate based nodes can be hardcoded, but anything that could change from node to node should be encoded into the metadata so that tools can take into account these differences.
Notably, the metadata does not contain any details around how to interact with the
state_call RPC method, which is the main thing that we are adding in V15 metadata.
Now, let’s talk about how metadata can be obtained from a node.
The existing runtime API essentially has this shape:
Metadata_metadata() -> frame_metadata::RuntimeMetadataPrefixed
This method can also be called directly via the
state_call RPC method, or when you make the RPC call
state_getMetadata it will be called under the hood for you. It takes no arguments, and returns some bytes that can decoded into
To accomodate updating to V15 metadata, the PR Metadata V15: Expose API to fetch metadata for version by lexnv · Pull Request #13287 · paritytech/substrate · GitHub adds two new runtime APIs which look something like:
// Fetch metadata at a specific version, None if no metadata for the given version:
Metadata_metadata_at_version(version: u32) -> Option<frame_metadata::RuntimeMetadataPrefixed>
// Fetch list of metadata versions that this node can provide:
Metadata_metadata_versions() -> Vec<u32>
Some notes about these:
- We now make explicit which version is being requested (so you don’t just get “whatever the latest” is). This helps us to maintain backward compatibility; nothing will break for people as new metadata versions are added over time, and Substrate can continue providing older versions. People can update to using newer versions when it suits them.
u32::MAXas a version number has been reserved to mean “the current unstable version”. Asking for this version on recent nodes will hand back our current unstable V15 metadata. This allows us to test and develop newer metadata versions “in the wild” without immediately having to commit to a stable format. ie, the unstable metadata handed back when
u32::MAXis used can change at any time.
Even though V15 metadata isn’t stable yet, these new Runtime APIs are, and can be used today on recent nodes (I think from runtime version 9420 or greater; already deployed on Westend but not Kusama/Polkadot yet at the time of writing). Over time, the desire is to deprecate and remove the “old”
Metadata_metadata interface and for people to use these new APIs.
Metadata_metadata will continue to serve V14 metadata always until it is eventually deprecated and removed.
Now you have some idea what the metadata is and how to obtain the metadata via the relevant runtime API, let’s talk about the changes we have made in V15 metadata.
The current changes that have been made for V15 metadata are the following:
- Metadata V15: Expose pallet documentation by lexnv · Pull Request #13452 · paritytech/substrate · GitHub - add pallet documentation to metadata.
- Metadata V15: Add Runtime API metadata by lexnv · Pull Request #13302 · paritytech/substrate · GitHub - add Runtime API information to metadata.
The latter is the most interesting change here. This PR adds to the metadata the information that we need to know which runtime API calls are available on some node, how to encode the parameters for them and how to decode the results that we get back. This is implemented already and can be made use of in Subxt today to make runtime API calls using a typesafe interface, albeit behind an “unstable-metadata” feature flag for now.
We’re going to look at adding two more small changes to V15 metadata before stabilizing it:
- Expose types for the overarching Call, Error, Event types · Issue #43 · paritytech/frame-metadata · GitHub - Expose Event and Error enums at the top level to make decoding these things simpler.
- V14 metadata doesn't provide enough information to decode extrinsic · Issue #12929 · paritytech/substrate · GitHub - Expose Address, Signature and Call types on the extrinsic details to allow for more reliable extrinsic decoding.
I’d like to note at this point that we don’t wish to be the bottleneck or gatekeepers for other changes that people would like to get into V15 metadata; our focus is on adding what we need to improve tooling like Subxt and CAPI. The approach we’ve taken here means that anybody who is interested in getting changes in to V15 metadata can raise issues/PRs in
frame-metadata to be discussed and implemented independently of ourselves. We’ll help out where we can though
- This V15 metadata is currently marked as unstable to give people a chance to add and test it before it is set in stone. I’d propose that we aim to stabilize V15 metadata by June 30th; this gives people (including ourselves) 1.5 months to push any final changes to it, try it out and raise issues/PRs if they would like to make additions. If anything significant comes up, we can postpone stabilization; my expectation is that it won’t.
- On or around June 30th, we will open PRs to shift the V15 metadata from being behind the unstable
u32::MAXversion to being behind version
15. At this point, V15 metadata will be stable, and no further changes can be made to it. Users and tools will then be able to rely on it going forwards, and any further changes will be made to a new V16-unstable metadata to be stabilized at a future date.
Note: V14 metadata will continue to be served as before from the old
Metadata_metadata Runtime API and
state_getMetadata RPC call, so nothing that relies on V14 metadata will suddenly break. Instead, people are gently encouraged to move to using the new Runtime APIs, and can continue to rely on V14 metadata as before, upgrading to V15 metadata only when it makes sense to them to do so.
I’m posting this to bring awareness to the upcoming change, but also to gather any feedback and hopefully try to answer any questions that come up