Metadata V16 updates

Coauthored-by: Pavel Khrystenko pavel.khrystenko:parity.io

Hello everyone,

We are excited to share an update on the progress of the upcoming v16 metadata. This update builds on our previous discussion, which you can find here: Upcoming Metadata v16 Features. For those interested in the technical details, you can track the implementation progress on the Polkadot SDK in this issue.

Key Highlights

Enrich metadata with associated types of config traits

This feature is designed to simplify the configuration process for tools like Subxt and PAPI when interacting with chains. By including associated types of Config traits in the metadata, we aim to reduce the manual configuration needed for these tools.

For instance, in Subxt, our goal is to leverage this metadata to automatically provide configurations for interacting with various Substrate-based chains. Currently, we manually define configurations for substrate and polkadot. This new feature aims to reduce the amount of manual config definition by inferring or deriving more of it from the metadata.

	#[pallet::config]
	pub trait Config: frame_system::Config {
		/// Associated type is now included in the metadata.
		type AccountId: TypeInfo;

		/// Associated type is now included in the metadata.
        type Address: TypeInfo;
    }

Developers can opt out of collecting metadata for associated types by using the without_metadata optional attribute in #[pallet::config].

Additionally, the without_metadata attribute can be used alongside the new #[pallet::include_metadata] attribute, allowing you to selectively include only certain associated types in the metadata collection.

For more details see PR: Metadata V16 (unstable): Enrich metadata with associated types of config traits

Extend metadata with deprecation information

To become aware of deprecated methods, developers must track a complex series of updates: from a Polkadot SDK release that deprecates a method, through various runtime upgrades and OpenGov referendums, and finally to the enactment of these changes

The feature aims to simplify this process by including deprecation information in the metadata. This will allow developers to easily identify deprecated methods and plan for their removal.

Parachain developers can include the deprecation information to the entire pallet, individual items, or even individual enum variants. This feature is particularly useful for dApp and tool developers.

    #[frame_support::pallet]
    /// Deprecate the entire pallet.
    #[deprecated(note = "Reason for deprecation")]
    pub mod pallet {

        #[pallet::config]
        pub trait Config: 'static {
            #[pallet::constant]
            /// Deprecate constants.
            #[deprecated(note = "Reason for deprecation")]
            type ExampleConstant: Get<()>;
        }
        
        #[pallet::event]
        /// Deprecate events.
        #[deprecated(note = "Reason for deprecation")]
        pub enum Event<T: Config>
        {
            ExampleA,
            /// Developers can also deprecate individual enum variants.
            /// #[deprecated(note = "Reason for deprecation")]
            ExampleB,
        }

        #[pallet::error]
        /// Deprecate events.
        #[deprecated(note = "Reason for deprecation")]
        pub enum Error<T: Config>
        {
            ExampleA,
            /// Developers can also deprecate individual enum variants.
            /// #[deprecated(note = "Reason for deprecation")]
            ExampleB,
        }

        #[pallet::call]
        impl<T: Config> Pallet<T>
        {
            #[pallet::call_index(0)]
            /// Deprecate calls.
            #[deprecated(note = "Reason for deprecation")]
            pub fn foo(
                origin: OriginFor<T>,
            ) -> DispatchResultWithPostInfo { }
        }

        #[pallet::storage]
        /// Deprecate storage items.
        #[deprecated(note = "test")]
        pub type OptionLinkedMap<T> = StorageMap<_, Blake2_128Concat, u32, u32, OptionQuery>;

    }

    sp_api::decl_runtime_apis! {
        /// Deprecate runtime APIs.
        #[deprecated]
        pub trait Api {
            #[deprecated(note = "example_method")]
            fn example_method();
        }
    }

For more info see PR: Deprecation info support in RuntimeMetadataIR

Next Steps

We are currently working on the implementation of these features and will provide updates as we progress. We encourage developers to follow the issues on the Polkadot SDK repository for more details.

We plan to introduce an Unstable Metadata to the frame-metadata crate and expose the unstable metadata via RPC calls. This will allow developers to experiment with the new metadata features before they are stabilized.

We are also encouraging external developers to provide feedback on the proposed changes. If you have any questions or suggestions, please feel free to reach out to us on in this post or on github.

(cc Polkadot-api team @josep @voliva @carlosala)

12 Likes

Awesome updates! Exposing deprecation information in the metadata is a really good addition. Right now without it, whenever a pallet call needs to be marked as deprecated, we have to use the workaround of adding a huge DEPRECATED warning on the doc comment in hopes someone downstream will read it. This will not only simplify dapps but also the FRAME codebase and process. Excited for it :grin:

2 Likes

What do you mean by this? Isnā€™t unstable metadata just u16::MAX that you can query using the metadata_at runtime call? I donā€™t really see any need to create any new RPC calls for this.

1 Like

Yep, weā€™ll not introduce a new RPC call, weā€™ll expose the u16::MAX via the metadata_at call. Thanks for pointing this out! :pray:

A great example of this will be the deprecation of Controllers, mostly being worked on by @ross.

The enum in the Rust code-base is marked as deprecated, and it would now be great if we see this deprecated message propagate outwards to PJS/PAPI:

2 Likes

Itā€™d be great as well to leak at the metadata the information about which hash function is used in the runtime to construct blocks, txs, etc. While in general this is Blake2b, other runtimes using Blake3 or Keccak (among others) could exist, and having this info would be useful for light nodes and clients like PAPI.

Iā€™m not sure if this needs special treatment in metadata v16 (as a compulsory field I mean) and it could of course be added as a constant in System pallet, for example. I envision it as part of the Config trait mentioned above.
Just something to think about.

1 Like

Thanks for the feedback everyone! :pray:

Itā€™d be great as well to leak at the metadata the information about which hash function is used in the runtime to construct blocks

By collecting the associated types weā€™ll also try to expose the Config::Hasher name into the metadata.

Another approach to this might be even to expose the hasher function code:

  • Joseph suggested adding a new runtime API call in this github issue
  • A more generic approach may be to expose code functionality directly by a dedicated wasm blob (Metadata V16: Add code functionality as WASM blob. As mentioned in the issue, there might be concerns about accidentally leaking private information to a malicious rpc node.

Iā€™ll look at the associated types collection and see if we expose enough information to deduce the hasher function

I think just having a runtime api for this is much easier. Just think about the hassle it would require to somehow include the precompiled WASM in your runtime WASM.

1 Like

This would be clearly enough. It is perfect if thereā€™s a common way to find it in the metadata as a name and a hasher available as a runtime API. Iā€™m concerned, though, about the potential DDOS attack on the nodes that would lead public RPCs to most likely block this kind of APIs. Having it on the metadata is crucial as well.

i dont think the Runtime Calls api is supported in v15, so might be useful to add those.

The metadata V15 exposes the Runtime API metadata. We currently use this information in Suxbt to generate a nicer API around those. Let me know if this is what you mean, thanks! :smiley:

1 Like

@lexnv indeed i also use the Runtime API to render all the metadata types, but the one thing that is missing is the ā€œruntime callā€ apis. It is there in the metadata however i dont know how to parse it correctly, im not sure its supported. Someone also left an open issue here about it.

this is what iā€™m referring to, you can find it in Runtime Calls in PJS apps, which i think is runtime call apis rather than runtime api, i believe.

Runtime calls are runtime apis. It is only pjs apps not using metadata V15. There are no metadata changes required. This is the issue about handling metadata V15 in pjs api Expose runtime methods from metadata V15 Ā· Issue #5725 Ā· polkadot-js/api Ā· GitHub

The work for PJS to dynamically produce the runtime apis via the metadata is ongoing. There has been some inherent issues I have found with initializing the API with metadata v15 which can be seen in this PR Initialize the API with metadata V15 by TarikGul Ā· Pull Request #5941 Ā· polkadot-js/api Ā· GitHub. Once that goes in it will be pretty trivial for me to add the runtime apis from the metadata instead of using the hardcoded augmented types.

1 Like

As an update, the runtime apis are now dynamically generated in the polkadot-js/api.

It will be fully reflected in next weeks release.

4 Likes

Please know that PAPI supports metadata v15 OOTB. Therefore, chain interactions for runtime APIs are available and strongly typed with PAPI. Also, dev-tools built with PAPI do support runtime-calls.

2 Likes

Really good updates!