XCM as a Standard for Reading And Interacting with Parachains

[shawntabrizi]

In discussions with Gav and Kian at the Cambridge Blockchain Academy, I was describing some of the pains that wallet developers face trying to program a single wallet application that works across multiple parachains, each with their own potential token pallet and API.

Gav quickly pointed out that this is yet another problem XCM solves.

You can have the wallet initiate a DOT transfer, directly on Polkadot, but using XCM to abstract away the specific pallet and apis underneath, rather than directly calling into the balances pallet as most wallets would do. In this way, this well formed XCM transfer call should work on any chain which supports XCM, independent of the pallets being used or anything like that.

Similarly, querying balances across all the chains is a big pain in the butt, and something that Kian can attest to, having made some apps that do this.

Instead, XCM again can be used to query a balance of a user, in some chain agnostic way. We would probably need to add some special XCM RPCs to properly support this scenario, but the point is that we do not actually need to require that each team implement a whole bunch of standards except for integration with XCM.

So in this case, we must ultimately see that XCM is not only about “cross-consensus” messages, but also messages within a single chain, and a way to talk to chains independent of the underlying implementation details.

NOTE: There is still a lot of work that we need to do in XCM to really support this vision end to end, but this is certainly the direction we all should be going, versus trying to standardize our parachain community at other layers, like pallets or rpcs or whatever. Standardizing at those layers would actually stifle growth and innovation of our products.

[kianenigma]

I can imagine an XCM-based runtime API that receives an input MultiAsset and an account id as input, and returns all of the assets belonging to that user account, and matches to that MultiAsset. If the MultiAsset is a wildcard, it would mean “give me all of the tokens belonging to this user”.

What I wonder is about the return type of this API. Should it still be the super generic XCM types like MultiAsset? probably no, and we’d only use MultiAsset as the query input. Instead, we need a more detailed return type that fulfills details such as:

• ticker
• decimals
• name
• …

Generally, I think abstracting balances behind XCM makes a lot of sense for transfers, but I am bit worried if it is reasonable for read-only operations. All we need is a standard return type and an API like:

fn tokens_of(who: AccountId) -> Vec<StandardBalanceType> {}

Not sure what values XCM can add to this (beyond the filtering, which I am not quite sure if it is actually sensible).

---

Regardless of with or without XCM: we can actually implement this first-hand on Polkadot. Historically, if you were to deposit your funds in crowdloans, it would be lost in your wallets. Nowadays, most wallets have implemented fix. Soon, nomination pools will have a very similar mechanism. Once you deposit funds into a pool, it will disappear from most wallets until they tediously fix it manually. Hopefully we can standardize this soon that nomination pools would be implemented in the more scalable way.

---

The TS-based project that I have started to tackle this in the short term is: GitHub - substrate-portfolio/polkadot-portfolio: Simple UI to find all your bags of tokens in the highly complicated world of Polkadot Ecosystem 🔴

[matt]

@shawntabrizi

NOTE: There is still a lot of work that we need to do in XCM to really support this vision end to end, but this is certainly the direction we all should be going, versus trying to standardize our parachain community at other layers, like pallets or rpcs or whatever. Standardizing at those layers would actually stifle growth and innovation of our products.

I love XCM being the layer that standardizes multi-chain interactions.

@kianenigma

What I wonder is about the return type of this API. Should it still be the super generic XCM types like MultiAsset? probably no, and we’d only use MultiAsset as the query input. Instead, we need a more detailed return type that fulfills details such as:

• ticker
• decimals
• name
• …

wen chainlink substrate pallet?

[IkerParity]

Very interesting idea indeed. This is also a recurring issue/point of discussion for the integration of, not only wallets, but also other external applications like custodians, exchanges etc.

Couple of considerations regarding:

You can have the wallet initiate a DOT transfer, directly on Polkadot, but using XCM to abstract away the specific pallet and apis underneath

Let’s assume that most of the wallets and exchanges in the ecosystem go for this approach:

• what would be the performance hit of this approach? I assume that doing a simple balance transfer will always be less complex than doing it via the XCM executor, would this be a negligible difference?

• Connected to the previous point, in terms of fees paid by the sender: how expensive do we expect balance transfers to be with this approach compared to direct balance transfers?

[kianenigma]

I brought this up again in a PR that I found to be solving something similar:

Return account's asset balances

Currently, an account can have “value bearing assets” in balances/assets pallet, in a DEX pallet, in some liquid staking pool pallet, in a lending platform. A wallet in our ecosystem has no meaningful way to scrape all of this information as it stands now, unless if they go on and study each parachain/platform one by one.
The XCM-centric approach would solve all of this in one go, and moves the responsibility of implementing this API to each individual chain, which is where it should logically reside.

[kckyeung]

I just had a thought while I was explaining Transact to someone else: knowing that in the XCM dialect issue, we attempt to decouple the RuntimeCall from the Transact instruction so that it can accept any arbitrary blob and interpret it accordingly, I was pondering that another approach to this problem would be to leave RuntimeCall in Transact, but change the definition of RuntimeCall altogether.

What I mean by this is that instead of defining the RuntimeCall as an amalgamation of the pallets’ extrinsics in the runtime, we could define it as the following:

enum RuntimeCall {
    Xcm(XcmCall),
    Extrinsic(ExtrinsicCall),
}

where ExtrinsicCall would retain the original definition of the RuntimeCall enum, but the XcmCall would then pretty much decode into a Vec<Instruction>, similar to how struct Xcm is defined, so that we can properly decode it and dispatch the necessary pallet calls to fetch the data that it needs. This coincidentally would also help with the implementation of XCM dialects, as we don’t need to create another struct or enum just to encode XCMs as the parameter to the Transact instruction.

Obviously, in such a setup, some XCM instructions will not be processed, as the runtime doesn’t (and shouldn’t) support all operations, but this is definitely a step towards allowing the runtime to interpret XCM as a general API. As I’m typing this, I also realized that we are essentially making the runtime an XCM executor as well, one that has a hugely reduced register count and functionality.

[gavofyork]

@kckyeung

we attempt to decouple the RuntimeCall from the Transact instruction so that it can accept any arbitrary blob and interpret it accordingly

Transact takes a blob already, and the semantics of the blob is entirely determined by the XCM implementation.

Beyond that, I’m not sure I understand either the proposal nor what it is meant to achieve. If you want to execute a bunch of XCM Instructions then you can already do that: just include them inline where the Transact instruction is.

[kckyeung]

@gavofyork

the semantics of the blob is entirely determined by the XCM implementation.

That unfortunately isn’t quite true yet, at least for some of the pallets that I’ve seen. Let’s take the XCM pallet as an example, the config item for XcmExecutor is defined thus:

		/// Something to execute an XCM message.
		type XcmExecutor: ExecuteXcm<<Self as SysConfig>::RuntimeCall>;

Since it uses the frame_system::RuntimeCall type as the type parameter, it means that the Transact bytes will only ever be interpreted as the aggregated runtime call type. The same situation happens in other pallets too, such as cumulus-pallet-xcm, dmp-queue and xcmp-queue.

The idea here is to allow the community to extend and evolve a set of instructions independently of the standard XCM set of instructions, and we call this an XCM dialect. In order for XCM dialects to work, we’d have to modify the type parameters in the configs of the aforementioned pallets and make them generic, instead of hard-coding frame_system::RuntimeCall.

With all that being said though, what my previous post was not about XCM dialect itself, but the possibility of combining both the XCM dialect and XCM-as-an-API together into a cohesive and systematic construct.

[bkchr]

@kckyeung

The idea here is to allow the community to extend and evolve a set of instructions independently of the standard XCM set of instructions, and we call this an XCM dialect. In order for XCM dialects to work, we’d have to modify the type parameters in the configs of the aforementioned pallets and make them generic, instead of hard-coding frame_system::RuntimeCall.

That is totally reasonable and should be a simple 50 lines changed pr that can be done now. Introduce some new TransactCall or whatever type to the Config trait of pallet-xcm and require that this type implements Dispatch + Encode + Decode. Then this can be mapped to RuntimeCall for now in the runtime and later it can be replaced by anything.

[gavofyork]

I don’t see why this should be introduced into the Transact instruction. You point out that one implementation of XCM (namely xcm-executor, which is very much opinionated towards FRAME) assumes that the byte blob in Transact decodes to a Call. It need not be the only implementation. A perfectly reasonable XCM implementation for a smart contract system might interpret Transact’s byte blob as an encoded smart contract call. The XCM specification (which does not opinionate itself toward FRAME, or indeed block-chains), explicitly stays away from defining the meaning of the byte blob of Transact.

While it’s perfectly possible (indeed trivial) to redefine how xcm-executor pallet interprets the Transact byte blob, I really don’t see what is to be gained by trying to do two completely different things with the same instruction. If we want an instruction which allows a programme in a separate language to be expressed, we should devise an instruction for it. There are questions over what elements of the XCVM should be exposed, how the language is defined, whether it should automatically subsume the overarching XCM executor and its instructions, and how different dialects might be combined, so I really don’t think it’s a trivial question to answer here. There may be better solutions, such as including some or more supported dialects within the version negotiation system, not entirely unlike OpenGL extensions.

[kckyeung]

My first intuition was indeed to add an explicit instruction for handling XCM dialects, but adding an instruction means that we would have to wait for the release of another XCM version, and I am not sure how long that would take.

The other aspect of this is to experiment whether using Transact is possible as a stop-gap solution with the current XCM version, until we can properly create a new instruction for it in the next. Some of the problems you’ve mentioned (e.g. combination of dialects, dialect discovery of a particular chain) were thought to be made possible to solve by simply defining a standard format of bytes in the Transact blob.

I don’t see a great need for dialects just yet, but it is certainly nice for parachain teams to be able to start thinking and developing a custom set of instructions that fit their particular needs. However, if it is cleaner and more appropriate for a new instruction to do so, then let’s add XCM dialects to the roadmap for the next XCM version, instead of trying to shoehorn the idea in the current XCM version.