XCM user and developer experience improvements

Tech Talk
21

I would like to provide an update on recent XCM UX improvement initiatives, looking for feedback from the community.

While each of the following sections/topics could have been its own Polkadot Forum thread, I decided to include them all here and suggest further discussions on the individual points happen on their linked Github pages. This will help keep all the feedback and information centralized.

NOTE: the items tagged XCMv5 can only be made available with a new (v5) XCM version, so will be delivered in the second half of this year. Everything else has either already landed or is under way, coming soon.

Addressing painpoint: XCM Fees

Estimating XCM fees

It is currently very difficult to figure out what are the total fees to be paid for XCM related operations. Total fees incurred by the complete execution of a cross-chain XCM program can be seen as the sum of:

  • local extrinsic tx fee (this is the classic tx fee and it is known or easy to find),
  • local XCM execution cost,
  • XCM delivery costs (for sending remote XCMs to other chains),
  • remote XCMs costs (remote execution + remote delivery) [repeated for each subsequent remote XCM].

In order to allow estimation of the total fees, we created XcmDryRun runtime APIs to expose the individual fees components on each chain along the XCM program path. An offchain entity (user, wallet, dApp component, etc) can simulate an extrinsic or XCM execution on the starting chain and repeat simulating the resulting remote XCMs on the subsequent chains to get an e2e overview of the total fees required as well as various other effects.

Furthermore, using the same API, emitted events across the cross-chain execution can be inspected for other purposes than just simulating fees.

You can see here multiple emulated asset transfer scenarios where the fees are “calculated off-chain” before executing the actual “on-chain transfer”.

Using multiple currencies for XCM fees

Chains that use pallet-asset-conversion (e.g. Asset Hub) could JIT swap any asset for DOT for the purpose of paying fees.

On Asset Hub, the fees required for local XCM execution have been integrated with above model, so that XCM execution can be
“bought” using any asset/currency that has a local DOT-pool set up. E.g. see this thread.

The (yet) missing piece is also hooking up XCM transport fees to the same mechanism. But this is also actively being worked on,
tracked by this issue, with a PR in progress.

This will allow e.g. fully paying fees using USDT.

Consolidating XCM execution and transport fees

While XCM execution fees are part of the XCM protocol and can be “bought” using BuyExecution instruction, XCM transport
fees only exist outside the XCM protocol and XCM programs have no control over them. This makes it very hard to create
complex or semi-complex reliable cross-chain XCM programs. In practice, many parachains have various hacks and workarounds
to be able to charge or handle these fees.

There is a new mechanism for specifying and paying for fees (execution and transport) in development, to be delivered with XCMv5.

I strongly urge anyone having had any sort of negative experience with the old model, to review the proposal and make sure their painpoints are addressed: RFC: Better fee handling #53 (XCMv5).

XCM cross-chain asset transfers

Hiding XCM complexity for non-trivial scenarios

While existing pallet_xcm::limited_teleport_assets(), pallet_xcm::limited_reserve_transfer_assets() could handle trivial asset transfers, they fell short when trying to combine multiple assets (with different reserves or teleport properties) in the same transfer. xtokens::transfer_multiassets() fares much better, but it also does not cover all scenarios.

pallet_xcm::transfer_assets() was added which allows mixing multiple asset types in same transfer with fewer artificial constraints imposed to the user.

You can see here an emulated e2e asset transfer between two parachains that teleports some asset while reserve-transferring another.

There is also now a higher layer JS asset-transfer-api library that offers wrappers over pallet-xcm and xtokens extrinsics which hides away even more XCM complexity.

I am hoping to see even more scenarios enabled in upper-layer libraries such as GitHub - paraspell/xcm-tools: ParaSpell✨: Set of XCM-Tools for Polkadot and Kusama ecosystem
(cc @dudo50).

Enabling complex scenarios

transfer_assets_using_type_and_then()

pallet_xcm::transfer_assets_using_type_and_then() was added for supporting complex multi-assets, multi-hops transfers.

For example moving KSM from “SomeKusamaParachain → Asset Hub Kusama → Asset Hub Polkadot → SomePolkadotParachain” in a single transfer (over the P<>K bridge). Or sending bridged ETH+DOT+USDT from “SomePolkadotParachain → Asset Hub Polkadot → OtherPolkadotParachain” while using a different asset for paying fees on each hop.

Example e2e scenarios are here (bridge transfers) and here (many hybrid transfers).

New XCM instructions

There’s an RFC up for new transfer instructions: RFC: add complex asset transfers instructions #54 (XCMv5).

It proposes new instructions that provide a way to initiate asset transfers which transfer multiple types (teleports, local-reserve, destination-reserve) of assets, on remote chains using XCM alone.

The currently existing instructions are too opinionated and force each XCM asset transfer to a single transfer type (teleport, local-reserve, destination-reserve). This results in inability to combine different types of transfers in single transfer which results in overall poor UX when trying to move assets across chains.

NOTE: these complex scenarios are now at least possible, but UX is poor for them, it requires heavy XCM knowledge to do them. Further work here to hide complexity is planned.

Better handling of transfer failures

Dedicated claim_assets extrinsic

pallet-xcm::claim_assets() was added to allow easier claim of trapped assets.

Fix UX for assets trapped on remote chains

New SetAssetClaimer instruction (XCMv5) to be added with XCMv5 which will allow XCM programs to set a custom Location as the authorized claimer of any assets trapped during XCM execution.

Idea under exploration and research

Overhaul of asset trapping mechanisms, use RemoteAccounts/SovereignAccounts when possible to deposit assets instead of trapping them.

General usability enhancements

Documentation and guides

Created an XCM Cookbook to explain XCM, as well as provide practical usage guides.

9 Likes
22

Thanks for mentioning our libraries @acatangiu. We in ParaSpell are very grateful for any mention.

We are working hard to keep up with recent changes in XCM and we also consult a lot with projects that are implementing our tools (To collect feedback, help them implement tools to their full potential and also act fast when the bug is found). As we (Me and my colleague) are now finishing our university studies we will be able to work even harder (full-time (As school has taken the majority of our time until now)).

As for features that we plan to implement ASAP:

claimAssets function (Along with proper guide)
Kusama<>Polkadot bridge
Multi-multilocation support (For multiple assets transportation within one message)

Other features that we wish to implement once they are released:
Snowbridge (Once active)
XCM v4 (Once more chains switch to the latest version) & v5 once out

What we have introduced recently:

XCM Analyser

  • Tool that allows for analysing XCM Multilocations and deconstructing them into a human-readable format (Created in cooperation with the PolkadotJS team for their Wallet extension (But has far more use case possibilities))

What we will introduce soon:

XCM Visualization tool

  • We had some data science lessons at our university and we found out, that there are no visualization tools for investors or data scientists (Who wish to explore this side of Polkadot also). The data can be extracted from Subscan but requires lots of effort. This will then be much easier, graphically laid out in 3D visualization (Small universe-like structure for each ecosystem) along with 2D visualizations(2D Graphs). (TBD within month/2)

There are many aspects to XCM that can be enhanced and we are super excited to delve into these. We believe this can greatly improve the experience for developers and users (As they enjoy smoothly working dapps).

I would also like to mention, that we offer free consultations to any project that wishes to implement any of our available XCM tool. Feel free to reach out to us via the contact form on our landing page.

With kind regards,
Team ParaSpell✨

2 Likes
23

Hey, @acatangiu, these are great steps to make XCM more usable.

  • I agree that transfer_multiasset is super helpful as it allows the transfer of reserve and non-reserve assets in a single XCM message
  • The issue of having multiple system chains that rely on DOT as a reserve is still a problem, tho, as parachains typically see only 1 reserve chain. I feel that the rebalancing mechanism of multiple sovereign accounts can serve as a stop-gap solution, but long term, we should find a better solution
  • I remember that with XCM V4, someone mentioned possibly doing asset transfers and transacting within a single XCM program. The main problem was that in asset transfers, you do ClearOrigin, while you need an origin for Transact. Is this still being worked on? It would be great as currently, if we want to do an asset transfer to be able to Transact, this requires 2 different XCM messages
  • I think the PolkadotXCM pallet needs to keep improving to abstract more of the complexities of XCM. For example, having functions that wrap some simple XCM programs like XCM Channel Open, Accept, Close, and remote transact. We at Moonbeam developed the XCM-Transactor Pallet, but having a more official implementation would benefit everyone. Especially for actual builders that build on top of XCM
  • I feel that Polkadot is missing big on exploiting (in the good sense) computed origins. This should be enabled in all Polkadot chains (relay, system chains and parachains) by default. Some time ago, there were already a lot of discussions on standards for account derivation when DescendOrigin was included. Therefore, if the functionality of remote transact is included in PolkadotXCM, we can actually start building programs in which users don’t have to go through fragmented experiences! Imagine that ANY user in Moonbeam can “natively” stake DOT directly on the Relay Chain or perform swaps in HydraDX, etc., without constantly switching wallets.
  • Lastly, building more robust use cases with Typescript/Javascript feels necessary as normally dApp builders use those languages to create user flows and applications. Having the Asset Transfer API is a great start, but creating examples with it, or of getting weight information via the new Runtime API call would be great
7 Likes
24

Some updates here targeted at the valuable suggestions from @albertov19:

The issue of having multiple system chains that rely on DOT as a reserve is still a problem, tho, as parachains typically see only 1 reserve chain.

With the “Minimal Relay” migration of balances to Asset Hub, Asset Hub becomes the main DOT reserve. Parachains can also use it as the only reserve and simply route DOT XCM transfers through Asset Hub.

possibly doing asset transfers and transacting within a single XCM program

This will be possible in XCMv5 with RFC#122 :tada:

I feel that Polkadot is missing big on exploiting (in the good sense) computed origins.

I agree! As discussed here, all system chains already have unified location → account conversion, all the using the same HashedDescription<AccountId, DescribeFamily<DescribeAllTerminal>> converter.

It’s now simply a matter of the ecosystem chains aligning on this too. I expect the chains that rely on cross-chain user actions have already or will quickly align. The others… well they probably don’t care or need it.

I think the PolkadotXCM pallet needs to keep improving to abstract more of the complexities of XCM. For example, having functions that wrap some simple XCM programs like XCM Channel Open, Accept, Close, and remote transact.

I actually envision leaner XCM pallets and richer TS/JS/Rust libraries and tools that build/expose these any many more flavors of XCM programs.

System chains now allow anyone to pallet_xcm::execute() and I am encouraging the rest of the ecosystem to do the same. This means we can build these useful XCM programs offchain, vend them as libraries and they can be used on any chain.
Developing these off-chain allows orders of magnitude improvements on: developers who could contribute, number of helpful programs provided, iteration speed, etc.

2 Likes
25

Yep, I completely agree with Bruno.

We need a playground for XCM. Not just an emulator; often, I don’t want to write code to test. Additionally, I’m tired of testing XCM commands in a testnet through Polkadot Apps; it’s too complicated.

I hope there will be a playground environment that can easily import the runtime configuration. This way, I can construct the XCM instructions to test. It would be even better if this playground could also provide a “debug trace stack,” similar to a general playground compiler debugger, so users can see what’s happening at each instruction.

It serves not only as a testing tool but also as an outstanding educational resource. With these capabilities, it can assist individuals in becoming acquainted with XCM more rapidly than ever before. You can’t deny this; one of the biggest problems of XCM is that it’s hard for newbies to learn. Compared to Ethereum, Polkadot is always more challenging to “get through the door.” Forget about the user for the moment; it’s even not very developer-friendly in some aspects.

2 Likes