Stablising V15 metadata

Tech Talk
1

Over the last few months, we in the Subxt team (well, mainly Alex :)) have introduced a V15 metadata format to Substrate and a couple of new Runtime API calls for accessing it. This V15 metadata format is currently unstable and subject to change. The aims of this post are to:

  • Summarize some of the key changes made as part of introducing this new version.
  • Mention the upcoming changes we’ll be making next.
  • Propose a date and approach for stabilizing V15 metadata.

What is metadata?

For those that don’t already know, the metadata obtained from a node is essentially a SCALE encoded blob that decodes to a frame_metadata::RuntimeMetadataPrefixed object (see frame-metadata/lib.rs at 438a5b098bb9d5b5a09bdc5b68275b2c5e63a010 · paritytech/frame-metadata · GitHub to see the exact type it decodes into).

The primary purpose of metadata (as I see it) is to provide all of the information necessary to be able to interact with a generic Substrate based node via its RPC APIs. It currently includes:

  • The information necessary to construct valid transactions that can be submitted via calls like author_submitExtrinsic.
  • The information necessary to decode errors and events that are returned from the node during extrinsic submission.
  • The information necessary to build valid storage keys and decode the storage items living at those locations.
  • The information necessary to access and decode any constant values declared in pallets.

Any data that is expected to be the same across all Substrate based nodes can be hardcoded, but anything that could change from node to node should be encoded into the metadata so that tools can take into account these differences.

Notably, the metadata does not contain any details around how to interact with the state_call RPC method, which is the main thing that we are adding in V15 metadata.

The Runtime API

Now, let’s talk about how metadata can be obtained from a node.

The existing runtime API essentially has this shape:

Metadata_metadata() -> frame_metadata::RuntimeMetadataPrefixed

This method can also be called directly via the state_call RPC method, or when you make the RPC call state_getMetadata it will be called under the hood for you. It takes no arguments, and returns some bytes that can decoded into frame_metadata::RuntimeMetadataPrefixed.

To accomodate updating to V15 metadata, the PR Metadata V15: Expose API to fetch metadata for version by lexnv · Pull Request #13287 · paritytech/substrate · GitHub adds two new runtime APIs which look something like:

// Fetch metadata at a specific version, None if no metadata for the given version:
Metadata_metadata_at_version(version: u32) -> Option<frame_metadata::RuntimeMetadataPrefixed>

// Fetch list of metadata versions that this node can provide:
Metadata_metadata_versions() -> Vec<u32>

Some notes about these:

  • We now make explicit which version is being requested (so you don’t just get “whatever the latest” is). This helps us to maintain backward compatibility; nothing will break for people as new metadata versions are added over time, and Substrate can continue providing older versions. People can update to using newer versions when it suits them.
  • u32::MAX as a version number has been reserved to mean “the current unstable version”. Asking for this version on recent nodes will hand back our current unstable V15 metadata. This allows us to test and develop newer metadata versions “in the wild” without immediately having to commit to a stable format. ie, the unstable metadata handed back when u32::MAX is used can change at any time.

Even though V15 metadata isn’t stable yet, these new Runtime APIs are, and can be used today on recent nodes (I think from runtime version 9420 or greater; already deployed on Westend but not Kusama/Polkadot yet at the time of writing). Over time, the desire is to deprecate and remove the “old” Metadata_metadata interface and for people to use these new APIs. Metadata_metadata will continue to serve V14 metadata always until it is eventually deprecated and removed.

Metadata V15 format

Now you have some idea what the metadata is and how to obtain the metadata via the relevant runtime API, let’s talk about the changes we have made in V15 metadata.

The current changes that have been made for V15 metadata are the following:

The latter is the most interesting change here. This PR adds to the metadata the information that we need to know which runtime API calls are available on some node, how to encode the parameters for them and how to decode the results that we get back. This is implemented already and can be made use of in Subxt today to make runtime API calls using a typesafe interface, albeit behind an “unstable-metadata” feature flag for now.

Upcoming changes

We’re going to look at adding two more small changes to V15 metadata before stabilizing it:

I’d like to note at this point that we don’t wish to be the bottleneck or gatekeepers for other changes that people would like to get into V15 metadata; our focus is on adding what we need to improve tooling like Subxt and CAPI. The approach we’ve taken here means that anybody who is interested in getting changes in to V15 metadata can raise issues/PRs in substrate or frame-metadata to be discussed and implemented independently of ourselves. We’ll help out where we can though :slight_smile:

Next steps

  1. This V15 metadata is currently marked as unstable to give people a chance to add and test it before it is set in stone. I’d propose that we aim to stabilize V15 metadata by June 30th; this gives people (including ourselves) 1.5 months to push any final changes to it, try it out and raise issues/PRs if they would like to make additions. If anything significant comes up, we can postpone stabilization; my expectation is that it won’t.
  2. On or around June 30th, we will open PRs to shift the V15 metadata from being behind the unstable u32::MAX version to being behind version 15. At this point, V15 metadata will be stable, and no further changes can be made to it. Users and tools will then be able to rely on it going forwards, and any further changes will be made to a new V16-unstable metadata to be stabilized at a future date.

Note: V14 metadata will continue to be served as before from the old Metadata_metadata Runtime API and state_getMetadata RPC call, so nothing that relies on V14 metadata will suddenly break. Instead, people are gently encouraged to move to using the new Runtime APIs, and can continue to rely on V14 metadata as before, upgrading to V15 metadata only when it makes sense to them to do so.

I’m posting this to bring awareness to the upcoming change, but also to gather any feedback and hopefully try to answer any questions that come up :slight_smile:

17 Likes
2

In addition to the newly introduced runtime API types in the metadata, there are plans to make some changes to the pallet’s metadata structure. Specifically, the PalletCallMetadata, PalletEventMetadata, and PalletErrorMetadata will be removed.

Previously, these structures included a pub ty: T::Type field, which identified the corresponding enum type for the pallet’s call, event, and error. However, this information will be directly available in the metadata V15 structure itself.

To accommodate this change, the following additions will be made to the RuntimeMetadataV15 structure:

pub struct RuntimeMetadataV15 {
...
	/// Metadata of the Runtime API.
	pub apis: Vec<RuntimeApiMetadata<PortableForm>>,

	/// The type of the outer `RuntimeCall` enum.
	pub call_enum_ty: <PortableForm as Form>::Type,
	/// The type of the outer `RuntimeEvent` enum.
	pub event_enum_ty: <PortableForm as Form>::Type,
	/// The type of the outer `RuntimeError` enum.
	pub error_enum_ty: <PortableForm as Form>::Type,
}

With these changes, the outermost enums will now include the previous pub ty: T::Type field to the enum variant corresponding to a given pallet.

The outermost enums are shaped similarly to:

enum RuntimeError {
    // Information regarding the System pallet that contains the
    // same type that was prior exposed by `PalletErrorMetadata`.
    System(frame_system::Error),
    ...
2 Likes
3

This is quite a “huge” change and it would be good to get some testing of people working on the other side :smiley:

1 Like
4

Subxt supports V14 and (unstable) V15 metadata at the moment, and I was wondering whether removing those PalletCallMetadata, PalletEventMetadata and PalletErrorMetadata might make this harder to do.

Ultimately though, the new outer variants added to the registry provide all of the old information and more, and so in Subxt we could either:

  • Keep using the pallet-specific type IDs in our Subxt metadata; we can extract them from these global variants anyway if handed V15 metadata to be compatible with how we’re doing things now, or
  • Benefit fully from these new outer enums, and if we’re handed V14 metadata we can manually create/append them to the scale_info::PortableRegistry given the information we already have.

ie it’s possible to upgrade or downgrade metadata still. (actually, that latter option is something we can look at right now, and might help us remove some special case things in our code anyway, so I might look into that!)

All in all, removing the types doesn’t make a huge difference to the total size (on Polkadot they account for ~0.2% the total metadata size if I can do maths), but generally trying to avoid duplicate information where possible seems like a good goal to me, so I’m in favour of this change!

I got thinking whether we could remove pallet names too, but I guess these new outer types may not have a variant for every pallet, so may be easier to keep them still…

5

Specifically, the PalletCallMetadata , PalletEventMetadata , and PalletErrorMetadata will be removed.

Why??

6

Because the information will then be duplicated in the metadata. If the size difference is really just ~0.2%, we could keep it, but it would still be duplicated info.

7

I see. Yeah 0.2% is not something worth a breaking change. We could for example in future want to compress the metadata, which should compress very well, and then there will be no duplicated info.

8

V15 is a breaking change any way, as new data is added and thus the encoding changes. So, we are already breaking it.

1 Like
9

As we’re working on a new deprecation process in FRAME I think that we the metadata could help easing the process of communicating deprecation and signature changes of extrinsics.

Currently, there is also no clear process to follow to deprecate or change the signature of extrinsics. Usually, we start by reaching out the community about the change and decide what is the best deprecation/update path. Communication is crucial since removing or refactoring extrinsics may break dapps (e.g. wallets with staking when the controller account is deprecated) and teams that do not keep up with the forum will only realise about the deprecation when the their app breaks.

One idea that has been floating around is to include in the metadata information about the extrinsics that are marked as deprecated and use that as a communication channel for projects to learn about soon-to-be deprecated or refactored extrinsics. The advantage is that community have an easier way to “subscribe” to changes in the extrinsics they depend, provided that they consume the metadata correctly. On the other hand, the disadvantage is that it bloats up the metadata, but the increase in storage requirements should be pretty low.

Another example of where this feature would be useful when we end up deprecating the chill_other extrinsic in staking, which we will postpone due to the potential breaking changes in the ecosystem apps. A feature like this would give us more confidence that we can do these types of changes with less friction.

Implementation ideas

  • We could annotate extrinsics with #[pallet::deprecate("substrate/issues/XXXX")] and then the metadata amalgamator extracts all the deprecation annotations and includes it in the extrinsic metadata with the issue referring with deprecation notes.

  • Another options would be only annotate with #[pallet::deprecate)] and rely on the extrinsic comments to add deprecation details and pointers.

Since Metadata V15 introduces breaking changes, perhaps this would be a good opportunity to include this feature. In any case, I wonder what is your opinion on this.

2 Likes
10

I have been thinking about the challenges we face while developing APIs in statically typed, compile-time languages such as C#. One issue that consistently arises is the inability to reuse generated code across different chains, even though they are using the same pallet version (we don’t know). This leads to a significant amount of redundant code in projects that handle multiple chains, such as explorers.

An approach that I believe could mitigate this issue involves incorporating pallet version information into the metadata of the chains. By having the metadata expose the version of the pallet being used, we can ensure that multiple chains using the same pallet version can reuse the same generated code. This could significantly reduce the amount of redundant code and streamline those projects.

Let me know if that could make sense, or if there is already a solution, like using the palletVersion from the storage somehow.

3 Likes
11

What is the preferred timeline for having the majority of people on V15? For example x% on y date.

12

I don’t have any real opinion on that; I expect we’ll want to keep the Runtime API calls to fetch V14 around for a long while yet, and so from my perspective it’s up to users when they want to update to fetching and benefitting from the added details in the V15 metadata. As with any piece of software, updating eventually is probably a wise move just to keep on top of things :slight_smile:

13

I understand. I was hoping to ascertain the impact v15 might have on the Swift type codegen referendum I have up for vote.

My existing work is based on the v14 metadata and I don’t have the bandwidth to diff the stabilized v15 right now. I was hoping to get a general timeframe. Would you be able to share adoption rates for previous metadata version releases?

14

Previous metadata version releases were essentially adopted as soon as chains updated to runtimes with the changes in (because the call to get metadata went from serving old to new version at that point). As such, there’s not really such a thing as an adoption rate beyond “how quickly did chains update to use the new runtime” (which I don’t have any answers to).

With V15 metadata, we made a couple of new runtime API calls and keep returning V14 from the old one, so it’s the first version that isn’t mandatory and can be adoped gradually (AFAIK).

I’d suggest that you don’t worry about rushing into using V15 metadata if it doesn’t provide anything of value. V14 metadata will continue to be provided, and so there’s no harm in continuing to make use of that.

The differences aren’t significant; most things remain very similar or identical between V15 and V14. Mostly we’ve just added some things, such as runtime API information, a custom field to allow chains to return arbitrary data, and a little more information about extrinsics.

For when you do have time to look, the differences can be seen by looking at https://github.com/paritytech/frame-metadata/blob/80ca5c2170928035e2adec9d2942a719086130fa/frame-metadata/src/v15.rs (it reuses V14 types for anything that hasn’t changed) and comparing it with https://github.com/paritytech/frame-metadata/blob/80ca5c2170928035e2adec9d2942a719086130fa/frame-metadata/src/v14.rs

I hope that helps!

1 Like