RFC: XCM Asset Transfer Program Builder

,

Following the enablement of arbitrary XCM execution on Kusama and Polkadot system chains, users/wallets/dapps can interact uniformly across multiple chains directly using XCM programs, without having to figure out chain-specific pallets or calls/extrinsics.

Improving Asset Transfer UX/DX

Cross-chain asset transfers are currently hard to generalize across the ecosystem. Each Parachain uses its own assets transfer pallet (usually xtokens or pallet-xcm) that exposes asset transfer extrinsics unique to that chain, and wallets/dapps need to integrate with all of them. These extrinsics then build opinionated XCM programs to do the asset transfers.

With the ability to execute arbitrary XCM programs, we can make use of this powerful layer of abstraction, and slowly transition to switching that around:

Wallets/dApps define their desired asset transfers in an XCM program and execute that using the “universal” pallet-xcm::execute() extrinsic. This has two major advantages:

  • Arbitrary XCM program (vs extrinsic) allows very high flexibility in what it can do. Current xtokens and pallet-xcm transfer extrinsics are limited to basic use-cases, and trying to expose more flexible ones results in ugly calls like pallet-xcm::transfer_assets_using_type_and_then().
  • Allows for high reusability across multiple chains. The same XCM program (save some relative location parameters) can be reused on other chains for the same purpose.

The disadvantage of running “raw” XCM programs is that they’re hard to build, it’s a low-level language with gotchas and corner-cases, and easy to get wrong. Calling an extrinsic to build and execute an XCM program for you is definitely simpler and more reliable.

But what if we could fix (or mitigate) that?

XCM Asset Transfer Tooling

For automated, safe, reliable building of a complex asset transfer XCM we will require multiple layers of “helper” tools/libraries each operating at a different level. A quick exercise of imagination produces the following top-down view:

  1. wallet/dApp: handles the “business usecase”: gets the user input and models “actions” based on it: some of these actions will be generic asset transfers like:

    • “(somehow) move AssetX from AccountA on ChainA to AccountB on ChainB”.
  2. some asset-transfer-library (ecosystem-stateful): has ecosystem-level knowledge/state, able to identify what cross-chain interactions are required to “move AssetX from ChainA to ChainB”, depending on the actual assets and chains involved it can define things like:

    • reserve location of AssetX,
    • cross-chain path from ChainA to ChainB (is it direct, through a reserve chain, over a bridge, etc?), and the “hops” involved
    • type of transfer of each asset between each hop (teleport, reserve-deposit, reserve-withdraw)

    libraries like asset-transfer-api and its assets-registry (cc @IkerParity)

  3. some XCM builder tool (ecosystem-stateless): Easy to use helper tool/library to hide away as much XCM complexity and be able to take the “detailed definition” of an asset transfer (involved accounts, assets, chains, transfer-types) and build a sanity-checked, corner-case-proofed, reliable XCM program for it.

This ^, alongside tooling that exposes the DryRun runtime APIs, should allow dApps to confidently switch to using flexible, portable, interoperable XCM programs instead of opinionated, chain-specific asset transfer extrinsics.

XCM Asset Transfer Builder

This RFC proposes the following (Rust) builder pattern tool for addressing layer (3) above. Same or similar can be created in JS or other relevant languages.

Builder properties:

  • sanity-checks any new “actions” (against the program built so far) handling gotchas and corner-cases,
  • “tag” assets in a given context (one chain) and use same tag to reference same asset in all other contexts (other chains) without having to explicitly manage XCM Locations context-relative views (no need to worry about reanchoring all the time),
  • hides away implementation details like how fees and assets might need separate transfers,
  • allows “natural”, in-order sequenced definition of actions across chains (vs the XCM program which will be a Matryoska doll that is built “backwards”, from the destination to the source),
  • enforces generated XCM correctness (assuming input/scenario-definition correctness).

Example 1 - HDX, USDT, GLMR from Hydration Network to Moonbeam (through AH)

fn example_hydra_to_ah_to_moonbeam() {
	// Initial context needs to start with `GlobalConsensus`
	let hydra_context = InteriorLocation::X2(GlobalConsensus(Polkadot), Parachain(HYDRA_PARAID));

	// used assets IDs as seen by the `hydra_context` chain
	let hdx_id = AssetId(Here.into());
	let usdt_id = AssetId(Location::new(
		1,
		X3(Parachain(ASSET_HUB_PARAID), PalletInstance(50), GeneralIndex(1984)),
	));
	let glmr_id = AssetId(Location::new(1, X1(Parachain(MOONBEAM_PARAID))));

	// Initialize the builder by defining the starting context and,
	let asset_transfer_builder = AssetTransferBuilder::using_context(hydra_context)
		// registering easy-to-use tags for the assets to transfer - caller can use these tags to
		// easily identify the assets without having to worry about reanchoring and contexts
		.define_asset("HDX", hdx_id)
		.define_asset("USDT", usdt_id)
		.define_asset("GLMR", glmr_id)
		// create the builder
		.create();

	let xcm = asset_transfer_builder
		// the starting context is Hydration Network
		// withdraw assets to transfer from origin account
		.withdraw("HDX", Definite(hdx_amount))
		.withdraw("USDT", Definite(usdt_amount))
		.withdraw("GLMR", Definite(glmr_amount))
		// set AssetHub as the destination for this leg of the transfer
		.set_next_hop(Location::new(1, X1(Parachain(ASSET_HUB_PARAID))))
		// teleport all HDX to Asset Hub
		.transfer("HDX", All, Teleport)
		// reserve-withdraw all USDT on Asset Hub
		.transfer("USDT", All, ReserveWithdraw)
		// reserve-withdraw all GLMR on Asset Hub
		.transfer("GLMR", All, ReserveWithdraw)
		// use USDT to pay for fees on Asset Hub (can define upper limit)
		.pay_remote_fess_with("USDT", Definite(max_usdt_to_use_for_fees))
		// "execute" current leg of the transfer, move to next hop (Asset Hub)
		.execute_hop()

		// from here on, context is Asset Hub
		// set Moonbeam as the destination for this (final) leg of the transfer
		.set_next_hop(Location::new(1, X1(Parachain(MOONBEAM_PARAID))))
		// reserve-deposit HDX to Moonbeam (note we don't need to worry about reanchoring in the new
		// context)
		.transfer("HDX", All, ReserveDeposit)
		// reserve-deposit USDT to Moonbeam (asset reanchoring done behind the scenes)
		.transfer("USDT", All, ReserveDeposit)
		// teleport GLMR to Moonbeam (asset reanchoring done behind the scenes)
		.transfer("GLMR", All, Teleport)
		// use GLMR to pay for fees on Moonbeam (no limit)
		.pay_remote_fess_with("GLMR", All)
		// "execute" current leg of the transfer, move to next hop (Moonbeam)
		.execute_hop()

		// from here on, context is Moonbeam
		// deposit all received assets to `beneficiary`
		.deposit_all(beneficiary)
		// build the asset transfer XCM Program!
		.finalize();

	// Profit!
	println!("Asset transfer XCM: {:?}", xcm);
}

Example 2 - KSM from Karura to Acala (over bridge, going through both Asset Hubs)

fn example_karura_to_acala() {
	// Initial context needs to start with `GlobalConsensus`
	let karura_context = InteriorLocation::X2(GlobalConsensus(Kusama), Parachain(KARURA_PARAID));

	// used assets IDs as seen by the `karura_context` chain
	let ksm_id = AssetId(Location::new(1, Here));

	// Initialize the builder by defining the starting context and,
	let asset_transfer_builder = AssetTransferBuilder::using_context(karura_context)
		// registering easy-to-use tags for the assets to transfer - caller can use these tags to
		// easily identify the assets without having to worry about reanchoring and contexts
		.define_asset("KSM", ksm_id)
		// create the builder
		.create();

	let xcm = asset_transfer_builder
		// the starting context is Karura
		// withdraw assets to transfer from origin account
		.withdraw("KSM", Definite(ksm_amount))
		// set Kusama AssetHub as the destination for this leg of the transfer
		.set_next_hop(Location::new(1, X1(Parachain(KUSAMA_ASSET_HUB_PARAID))))
		// reserve-withdraw all KSM on Asset Hub
		.transfer("KSM", All, ReserveWithdraw)
		// use KSM to pay for fees on Kusama Asset Hub (no limit)
		.pay_remote_fess_with("KSM", All)
		// "execute" current leg of the transfer, move to next hop (Kusama Asset Hub)
		.execute_hop()

		// from here on, context is Kusama Asset Hub
		// set Polkadot Asset Hub as the destination for this leg of the transfer
		.set_next_hop(Location::new(
			2,
			X2(GlobalConsensus(Polkadot), Parachain(POLKADOT_ASSET_HUB_PARAID)),
		))
		// reserve-deposit KSM to Polkadot Asset Hub (asset reanchoring done behind the scenes)
		.transfer("KSM", All, ReserveDeposit)
		// use KSM to pay for fees on Polkadot Asset Hub (no limit)
		.pay_remote_fess_with("KSM", All)
		// "execute" current leg of the transfer, move to next hop (Polkadot Asset Hub)
		.execute_hop()

		// from here on, context is Polkadot Asset Hub
		// set Acala as the destination for this leg of the transfer
		.set_next_hop(Location::new(1, X1(Parachain(ACALA_PARAID))))
		// reserve-deposit KSM to Acala (asset reanchoring done behind the scenes)
		.transfer("KSM", All, ReserveDeposit)
		// use KSM to pay for fees on Acala (no limit)
		.pay_remote_fess_with("KSM", All)
		// "execute" current leg of the transfer, move to next hop (Acala)
		.execute_hop()

		// from here on, context is Acala
		// deposit all received assets to `beneficiary`
		.deposit_all(beneficiary)
		// build the asset transfer XCM Program!
		.finalize();

	// Profit!
	println!("Asset transfer XCM: {:?}", xcm);
}

Github issue/discussion: RFC: XCM Asset Transfer Program Builder · Issue #4736 · paritytech/polkadot-sdk · GitHub

cc @xlc @francisco.aguirre

2 Likes

Kudos of this proposal, I believe it is the right way to enable the ecosystem-wide potential of XCM.

Couple of considerations:

  • Should the execute() approach replace all the ad-hoc extrinsics in pallet_XCM or xTokens pallet in the long term? In terms of runtime maintenance/cleanliness this makes sense, but it may be a considerable pain for APIs and wallets supporting the current interfaces.
  • Is this proposal limited to transfers for any specific reason? I think more complex interactions (like remote locking) can flow naturally from such a tool.
  • is the role of the upper layer (asset-transfer-library) limited to provide the context (ecosystem stateful)? How would it interact with the builder? Would it mainly assign values to the context variables in the examples?

That being said, looking forward to supporting this proposal and collaborating in the development of the builder tool and the asset-transfer-library.

1 Like

Hey Iker, thanks for the feedback!

Should the execute() approach replace all the ad-hoc extrinsics in pallet_XCM or xTokens pallet in the long term? In terms of runtime maintenance/cleanliness this makes sense, but it may be a considerable pain for APIs and wallets supporting the current interfaces.

I hope so, yes! But it will not be forced or hurried, existing APIs, wallets and dApps can continue using the existing extrinsics in the short and medium term. My expectation is that they will use the execute() model for new usecases that are hard to support otherwise, and over time also organically move the simple usecases as well for consistency and easier maintenance on their side.

Is this proposal limited to transfers for any specific reason? I think more complex interactions (like remote locking ) can flow naturally from such a tool.

I am personally in favor of simple tools that do one thing only but do it well (and easy). We already have complex swiss army knives at most layers of our stack and high complexity is one of the painpoints of our ecosystem.
As such, I was thinking for these builder/helper tools/crates/libs to be very mindful of what usecases we add to them. Maybe we can have more of them, specialized ones, more general ones, etc. We’ll see as we go along and the ecosystem needs become apparent.

is the role of the upper layer (asset-transfer-library) limited to provide the context (ecosystem stateful)? How would it interact with the builder? Would it mainly assign values to the context variables in the examples?

The 3 layer example was just a quick idea, we may have more, we may have less.
E.g. you could use such a builder from a 3rd party or develop this builder internally as native part of asset-transfer-api or skip it altogether and use raw XCM, I don’t think there is a right or wrong design.

The point was to have something to hide some of the complexity and gotchas of doing raw XCM programs. I am proposing this “builder” layer as a reusable component to potentially accelerate/help any dApp/wallet/lib currently doing raw XCMs.

1 Like

All this makes sense, thanks!

I also see now the benefit of having first this XCM builder focused on building XCM programs for transfers and then consider extending it or implementing a new one of other cross-chain use cases.

As I said in the prev comment, we are going to look closer to this in the next weeks and see how to better support the realization of this vision :handshake: