XCM as a Standard for Reading And Interacting with Parachains

[shawntabrizi]

In discussions with Gav and Kian at the Cambridge Blockchain Academy, I was describing some of the pains that wallet developers face trying to program a single wallet application that works across multiple parachains, each with their own potential token pallet and API.

Gav quickly pointed out that this is yet another problem XCM solves.

You can have the wallet initiate a DOT transfer, directly on Polkadot, but using XCM to abstract away the specific pallet and apis underneath, rather than directly calling into the balances pallet as most wallets would do. In this way, this well formed XCM transfer call should work on any chain which supports XCM, independent of the pallets being used or anything like that.

Similarly, querying balances across all the chains is a big pain in the butt, and something that Kian can attest to, having made some apps that do this.

Instead, XCM again can be used to query a balance of a user, in some chain agnostic way. We would probably need to add some special XCM RPCs to properly support this scenario, but the point is that we do not actually need to require that each team implement a whole bunch of standards except for integration with XCM.

So in this case, we must ultimately see that XCM is not only about “cross-consensus” messages, but also messages within a single chain, and a way to talk to chains independent of the underlying implementation details.

NOTE: There is still a lot of work that we need to do in XCM to really support this vision end to end, but this is certainly the direction we all should be going, versus trying to standardize our parachain community at other layers, like pallets or rpcs or whatever. Standardizing at those layers would actually stifle growth and innovation of our products.

[kianenigma]

I can imagine an XCM-based runtime API that receives an input MultiAsset and an account id as input, and returns all of the assets belonging to that user account, and matches to that MultiAsset. If the MultiAsset is a wildcard, it would mean “give me all of the tokens belonging to this user”.

What I wonder is about the return type of this API. Should it still be the super generic XCM types like MultiAsset? probably no, and we’d only use MultiAsset as the query input. Instead, we need a more detailed return type that fulfills details such as:

• ticker
• decimals
• name
• …

Generally, I think abstracting balances behind XCM makes a lot of sense for transfers, but I am bit worried if it is reasonable for read-only operations. All we need is a standard return type and an API like:

fn tokens_of(who: AccountId) -> Vec<StandardBalanceType> {}

Not sure what values XCM can add to this (beyond the filtering, which I am not quite sure if it is actually sensible).

---

Regardless of with or without XCM: we can actually implement this first-hand on Polkadot. Historically, if you were to deposit your funds in crowdloans, it would be lost in your wallets. Nowadays, most wallets have implemented fix. Soon, nomination pools will have a very similar mechanism. Once you deposit funds into a pool, it will disappear from most wallets until they tediously fix it manually. Hopefully we can standardize this soon that nomination pools would be implemented in the more scalable way.

---

The TS-based project that I have started to tackle this in the short term is: GitHub - substrate-portfolio/polkadot-portfolio: Simple UI to find all your bags of tokens in the highly complicated world of Polkadot Ecosystem 🔴

[IkerParity]

Very interesting idea indeed. This is also a recurring issue/point of discussion for the integration of, not only wallets, but also other external applications like custodians, exchanges etc.

Couple of considerations regarding:

You can have the wallet initiate a DOT transfer, directly on Polkadot, but using XCM to abstract away the specific pallet and apis underneath

Let’s assume that most of the wallets and exchanges in the ecosystem go for this approach:

• what would be the performance hit of this approach? I assume that doing a simple balance transfer will always be less complex than doing it via the XCM executor, would this be a negligible difference?

• Connected to the previous point, in terms of fees paid by the sender: how expensive do we expect balance transfers to be with this approach compared to direct balance transfers?

[kianenigma]

I brought this up again in a PR that I found to be solving something similar:

Return account's asset balances

Currently, an account can have “value bearing assets” in balances/assets pallet, in a DEX pallet, in some liquid staking pool pallet, in a lending platform. A wallet in our ecosystem has no meaningful way to scrape all of this information as it stands now, unless if they go on and study each parachain/platform one by one.
The XCM-centric approach would solve all of this in one go, and moves the responsibility of implementing this API to each individual chain, which is where it should logically reside.

[kckyeung]

I just had a thought while I was explaining Transact to someone else: knowing that in the XCM dialect issue, we attempt to decouple the RuntimeCall from the Transact instruction so that it can accept any arbitrary blob and interpret it accordingly, I was pondering that another approach to this problem would be to leave RuntimeCall in Transact, but change the definition of RuntimeCall altogether.

What I mean by this is that instead of defining the RuntimeCall as an amalgamation of the pallets’ extrinsics in the runtime, we could define it as the following:

enum RuntimeCall {
    Xcm(XcmCall),
    Extrinsic(ExtrinsicCall),
}

where ExtrinsicCall would retain the original definition of the RuntimeCall enum, but the XcmCall would then pretty much decode into a Vec<Instruction>, similar to how struct Xcm is defined, so that we can properly decode it and dispatch the necessary pallet calls to fetch the data that it needs. This coincidentally would also help with the implementation of XCM dialects, as we don’t need to create another struct or enum just to encode XCMs as the parameter to the Transact instruction.

Obviously, in such a setup, some XCM instructions will not be processed, as the runtime doesn’t (and shouldn’t) support all operations, but this is definitely a step towards allowing the runtime to interpret XCM as a general API. As I’m typing this, I also realized that we are essentially making the runtime an XCM executor as well, one that has a hugely reduced register count and functionality.

[gavofyork]

@kckyeung

we attempt to decouple the RuntimeCall from the Transact instruction so that it can accept any arbitrary blob and interpret it accordingly

Transact takes a blob already, and the semantics of the blob is entirely determined by the XCM implementation.

Beyond that, I’m not sure I understand either the proposal nor what it is meant to achieve. If you want to execute a bunch of XCM Instructions then you can already do that: just include them inline where the Transact instruction is.

[kckyeung]

@gavofyork

the semantics of the blob is entirely determined by the XCM implementation.

That unfortunately isn’t quite true yet, at least for some of the pallets that I’ve seen. Let’s take the XCM pallet as an example, the config item for XcmExecutor is defined thus:

		/// Something to execute an XCM message.
		type XcmExecutor: ExecuteXcm<<Self as SysConfig>::RuntimeCall>;

Since it uses the frame_system::RuntimeCall type as the type parameter, it means that the Transact bytes will only ever be interpreted as the aggregated runtime call type. The same situation happens in other pallets too, such as cumulus-pallet-xcm, dmp-queue and xcmp-queue.

The idea here is to allow the community to extend and evolve a set of instructions independently of the standard XCM set of instructions, and we call this an XCM dialect. In order for XCM dialects to work, we’d have to modify the type parameters in the configs of the aforementioned pallets and make them generic, instead of hard-coding frame_system::RuntimeCall.

With all that being said though, what my previous post was not about XCM dialect itself, but the possibility of combining both the XCM dialect and XCM-as-an-API together into a cohesive and systematic construct.

[bkchr]

@kckyeung

The idea here is to allow the community to extend and evolve a set of instructions independently of the standard XCM set of instructions, and we call this an XCM dialect. In order for XCM dialects to work, we’d have to modify the type parameters in the configs of the aforementioned pallets and make them generic, instead of hard-coding frame_system::RuntimeCall.

That is totally reasonable and should be a simple 50 lines changed pr that can be done now. Introduce some new TransactCall or whatever type to the Config trait of pallet-xcm and require that this type implements Dispatch + Encode + Decode. Then this can be mapped to RuntimeCall for now in the runtime and later it can be replaced by anything.

[gavofyork]

I don’t see why this should be introduced into the Transact instruction. You point out that one implementation of XCM (namely xcm-executor, which is very much opinionated towards FRAME) assumes that the byte blob in Transact decodes to a Call. It need not be the only implementation. A perfectly reasonable XCM implementation for a smart contract system might interpret Transact’s byte blob as an encoded smart contract call. The XCM specification (which does not opinionate itself toward FRAME, or indeed block-chains), explicitly stays away from defining the meaning of the byte blob of Transact.

While it’s perfectly possible (indeed trivial) to redefine how xcm-executor pallet interprets the Transact byte blob, I really don’t see what is to be gained by trying to do two completely different things with the same instruction. If we want an instruction which allows a programme in a separate language to be expressed, we should devise an instruction for it. There are questions over what elements of the XCVM should be exposed, how the language is defined, whether it should automatically subsume the overarching XCM executor and its instructions, and how different dialects might be combined, so I really don’t think it’s a trivial question to answer here. There may be better solutions, such as including some or more supported dialects within the version negotiation system, not entirely unlike OpenGL extensions.

[kckyeung]

My first intuition was indeed to add an explicit instruction for handling XCM dialects, but adding an instruction means that we would have to wait for the release of another XCM version, and I am not sure how long that would take.

The other aspect of this is to experiment whether using Transact is possible as a stop-gap solution with the current XCM version, until we can properly create a new instruction for it in the next. Some of the problems you’ve mentioned (e.g. combination of dialects, dialect discovery of a particular chain) were thought to be made possible to solve by simply defining a standard format of bytes in the Transact blob.

I don’t see a great need for dialects just yet, but it is certainly nice for parachain teams to be able to start thinking and developing a custom set of instructions that fit their particular needs. However, if it is cleaner and more appropriate for a new instruction to do so, then let’s add XCM dialects to the roadmap for the next XCM version, instead of trying to shoehorn the idea in the current XCM version.

[kckyeung]

It’s becoming clear to me that XCM by itself now couldn’t actually express the idea of querying balances. After some more discussions with Gav, the primitives are already in place, but the language to query is not.

For a concrete example, let’s say we have a query instruction that has a syntax of BalanceOf(who, asset). who in this context can already be represented as a MultiLocation, and so does asset with MultiAsset, but what is missing here is the ability to expressBalanceOf – none of the XCM instructions so far is able to express this idea.

The most straightforward solution is to introduce a new XCM instruction to that effect, and while it is true we can do so, we should take a step back and look at the XCM executor in principle:

• It does not natively support the notion of a “result”; in other words, XCM instructions aren’t expressions that always return a value after execution, and this is incompatible with the concept of queries as they do return a result by definition;
• Queries are not mutative, but XCM instructions cannot guarantee this quality, as it is designed with known side effects;
• BalanceOf is an example of a query, but the entire universe of possible queries is a space where the computational complexity is not quite yet known, and can possibly be large, or even unconstrained, and can spell problems for on-chain applications.

Owing to the reasons above, it would make more sense to create a new XCQ language for querying purposes, the prototype of which could deal with simple queries and leave the support for complex queries for later.

[xlc]

I already have proposals to solve the query issue on different context:

Onchain:

Interchain Proof Oracle Network

Background XCM provides a trustless way for relay/parachains to dispatch messages to each other. However, there is another common use case that’s not fully addressed by XCM, which is the ability to trustlessly read onchain state from another chain. While it is possible to create a protocol on top of XCM to allow chains to publish their onchain states, it may not necessary be the best way due to overheads of XCM. Especially at this stage we are still using XCM-lite that all the messages consume…

Offchain:

Wasm view functions

Background In Solidity, contract developers can create view functions for their contract to help clients to access contract state. In Substrate, we have runtime APIs that achieve similar goal. That’s totally useable and a good enough solution. However, it still have a lot of room for improvement. One of the nice property of the Solidity view function is that the same getter can be used for both contract logic and front-end and that reduces a lot of duplicated codes. For example, a rebase toke…

[kianenigma]

Can you expand a bit further on how and why this is so interlinked to xcm-executor?

My understanding was that, at least in the first phase, we use XCM for this goal only to leverage the MultiAsset type as a generic representation of an asset. I envisioned this initially as a runtime-api (see my comment above), or as @xlc said, a view function. I think all of this can be done, and the implementation can be left to each chain’s runtime, without the need to involve XCVM aka the xcm-executor. This, combined with a global asset registry (which is also in the work by the community, see linked threads) is already a big reduction of burden on wallets, with minimal overhead for parachai=ns, with very little effort.

In other words, each parachain spends a bit of time to implement this query, and then all wallets are free from needing to re-implement the same thing. Then all DotSama wallets can show your assets in all parachains much easier.

I am worried that I have been linking to this thread mistakenly and we talking about two different things here.

To expand further, assuming that all almost all chains have custom pallets that hold some value bearing MultiAsset that you wish to return as a part of the return type of either of BalanceOf(who, asset) or tokens_of(who: AccountId) -> Vec<MultiAsset>, there is no way around leaving (at least part of) the implementation of this query to the chain’s logic. In other words, I can’t see any way in which xcm-executor can solely implement this. It has to be partially configurable by

[shawntabrizi]

@kckyeung

Owing to the reasons above, it would make more sense to create a new XCQ language for querying purposes, the prototype of which could deal with simple queries and leave the support for complex queries for later.

I think this is what i mean, just without creating “another language”.

XCM has all the primitives needed to make a query language, such as Multiassets, versioning, etc… I am not suggesting we add a new XCM instruction (although maybe it ends up being like this, but uncallable from the runtime), but simply a new RPCs (which forward to the runtime apis as @kianenigma said), which understands XCM primitives, queries the state, and can return data using them.

For example, the only state query right now is state.getStorage(key), for which you provide a key and it returns raw bytes. I suggest, as a first step, creating a new RPC which is xcm.queryBalances(account) which you provide an AccountId, and it returns a Vec<Multiasset>. Again, this is a repeat of a lot of what @kianenigma said.

I am wrapping this all under “XCM” because the primitives of XCM should have no problem handling these kinds of things, and it could have its own set of RPCs to do all we need here.

[bkchr]

@shawntabrizi

For example, the only state query right now is state.getStorage(key), for which you provide a key and it returns raw bytes. I suggest, as a first step, creating a new RPC which is xcm.queryBalances(account) which you provide an AccountId, and it returns a Vec<Multiasset>. Again, this is a repeat of a lot of what @kianenigma said.

Statemine and Statemint already provide this kind of api: https://github.com/paritytech/cumulus/blob/321e81e4eeb5559df336581cb6819c7edf814a2d/parachains/runtimes/assets/common/src/runtime_api.rs#L44

[tomaka]

I don’t know much about XCM, but I’d like to weight in on the topic of adding RPC functions to remind that adding RPC functions that need to be aware of some details of the runtime (such as some type definitions or the presence of a certain function with a certain signature) is fundamentally wrong.

If the runtime upgrades to a new version that changes some of these details, the RPC function will be utterly broken on all the nodes that haven’t upgraded.
Not only there is no way to provide a smooth transition during a runtime upgrade, it also forces every RPC node operator to upgrade as quickly as possible.
Instead, the logic of all the RPC functions that aren’t low level should be moved to the JSON-RPC client side, usually the JS library.

I understand that you all are Rust developers and you like to write things in Rust, but that’s IMO not a good enough reason to push for a broken design.

Pinning RPC functions to the details of the runtime also goes in the opposite direction of splitting the code of the runtime away from the client, which as far as I’m aware is still an objective.

[shawntabrizi]

I generally agree with @tomaka’s philosophy here.

Perhaps what I want needs to live at a different level of the abstraction stack, but I also think it is possible you are interpreting XCM to be something it is not.

I think it is also fair to say that XCM, as a query language, is perfectly general to all kinds of blockchains. It does not need to actually expose any of the specifics from the runtime, just as XCM executor does not assume anything about the specific implementation of the runtime either. Instead, it has only made assumptions about the kinds of operations and instructions which are handled by “consensus systems”.

It may be that a blockchain has no currencies on the chain, and for such a query, the result could be None. But for such a query to exist in general seems perfectly fine. And if done correctly, it can be entirely up to the runtime on how the result is reported.

[gavofyork]

I had a much larger conversation (which Keith is referencing) about this topic but unfortunately it’s in the Parity XCM channel.

Basically, we need a means of querying some information encoded within a chain’s state. This system must:

• not require computation to be done on the target chain
• work as well when the querying system is on-chain (ie within a runtime or smart contract) as well as if it’s off-chain (eg a script or light client)
• work just as well for any chain which can provide some basic metadata (much like XCM works well for any chain which contains an implementation of an XCM executor)
• be extensible and work across a variety of use cases not just the obvious ones right now (eg balance querying)

XCM contains some useful primitives as Shawn says like MultiLocation and MultiAsset. However it is ultimately a scripting language for mutating state. It is not a query language. Retrofitting querying primitives would convolute it at basically zero gain. It’s conceivable that XCQ could be a linguistic subset of XCM, but I think it’s not necessary to determine that at this point.

What we (may soon) have is a means of querying one or more values in a chain’s state, and assuming we have ground truth on the state’s merkle root then have certainty that these values are genuine.

However the meaning of these values is in general unknown. XCQ can be considered a means of giving these values meaning, and to determine which keys’ values are needed for any given meaning to be determinable. In Star Trek terms, it’s a code sheet for the universal translator, to both tell us what to actually ask for from state for any given piece of information to be determined and to translate this witness data into our final result.

It lets chains (which have co-knowledge of each other’s state roots via e.g. a bridge or shared relay) understand the information within each other’s state and it lets scripts (which have knowledge of a target chain’s state root e.g. by Smoldot) to utilise the information held within the chain’s state; and, crucially, it does so without any kind of operational requirement on the chain being queried. In particular there’s no need to add RPCs or use any particular software. It’s fully compatible with the information-logistics model provided by Smoldot.