XCM is a crucial part of Polkadot. It lets the heterogeneous parachains communicate with each other and become more efficient by relying on others for certain things they are not great at.
However, most messages are transfers (1), which are just one of the things that can be done.
I feel adoption and innovation is lacking, mainly because of bad developer and user experience.
One thing that is needed is examples, so I’ll be working on extending the xcm-docs we released earlier this year.
Some ideas were already expressed in XCM docs suggestions, but please do share more suggestions.
Another thing needed is making the API friendlier.
I ask for your ideas and feedback to improve this aspect.
Here are some, which you can also comment on:
One aspect to consider and a reason I brought up something like XMS is how can we make XCM shine more/be simpler in the client side. pallet-xcm’s send/execute cover a lot of ground already, so if instead of focusing on Rust and wrapper pallets we provide a simpler and hopefully language-agnostic-declarative way to write composable XCM then we could bring new developers and kinds of use-cases.
Well, lets step back and look this from a 10000 foot view and searching for bottlenecks first.
I see a few things are blocking
Security consideration. It is unsafe to allow user to craft and send arbitrary XCM. It can be unsafe to execute arbitrary XCM. There are some issues and PR to improve this but well, they are not yet closed. On top of that, it is too easy to create an unsafe XCM configuration. Yeah docs are helpful but it will be better if we can leverage compiler to prevent unsafe configurations. I don’t have a lot of idea on if that’s possible at all, but surely there are areas of improvements.
Standards. XCM covers simple thing like transfer but for more advanced tasks, like crosschain smart contract invocation, we need to use Transact and that just bad. Again, there were work on this area but not yet completed.
I don’t really have solution of even suggestions, other than there are still a lot of work remaining. If you are unsure what to do next, well, how about check the issues that are opened for years.
By definition, standards need to be created and agreed upon by the community. It is not particularly wise to depend upon a single entity like Parity to come up with solutions for each and every business use-case. The fact that we don’t see much discussions about standards, either on this forum or elsewhere, is part of the problem that we hope to solve with better API and documentation. No discussion = no standards = no innovation.
Hence, the bottleneck from my view is a lack of a simple and easily digestible language to talk about XCM in general. One cannot even begin any sort of discussion on a topic that they don’t have a good mental model of. What we at Parity can do is to provide the tools and primitives to tinker around and play with XCM, but the future of how XCM evolves must be community-led. This is the raison d’être behind the XCM RFC process.
A reasonable suggestion is to first look to your right and left, and explore what is done in other ecosystems.
Eg. what are the top most common interactions done in other multichain ecosystems? or even bridges between sovereign ecosystems? (I think bridges are probably almost always just asset transfers, but still worth exploring).
Once you have a few examples like this, we can see how easy it is to do the same in XCM and in Polkadot. If it is overly difficult, there is a conversation to be had of why and how it can be improved.
This is step 0. Then, once we are well grounded in this foundation, we can think of what use-cases are then particularly shitty in other ecosystems that lack shared security, and then see if/how Polkadot’s shared security would improve that scenario. Then XCM should support that (hopefully killer) use case in an “as simple as possible” manner. I hope this mindset can lead to innovation.
Well, we already have the Rust compiler which is good at preventing people from doing things they are not suppose to do. I guess it would be possible to redesign the types of XCM configs to catch some errors. However, this is just hard.
Take an example, fees. It is unsafe if a parachain don’t charge fees for XCM, well, usually. That means we want to prevent the code from compiling if the chain did not configure a fee deduction mechanism for XCM processing. Except we basically can’t validate at compile time to determine a config is safe or not. There are always exceptions. System parachains don’t charge fees from governance initiated XCM from relaychain and that’s safe.
Validating XCM configs for safeness is an interesting idea, I think something could definitely be done. Sane defaults would be a good start.
One idea regarding fees, not on the config side, but on the message creating side, is taking advantage of the builder pattern to disallow creating messages that don’t pay for fees for example.
Something like:
let paying_fees: Xcm<()> = Xcm::builder() // Only allow paying for fees
.withdraw_asset() // First instruction has to load the holding register
.buy_execution() // Second instruction has to be `buy_execution`
.build();
let paying_fees_invalid: Xcm<()> = Xcm::builder()
.withdraw_asset()
.build(); // Invalid, need to pay for fees
let not_paying_fees: Xcm<()> = Xcm::builder_unpaid()
.unpaid_execution() // Needed
.withdraw_asset()
.deposit_asset()
.build();
let explicitly_without_fees: Xcm<()> = Xcm::builder_unsafe() // You can do anything
.withdraw_asset()
.deposit_asset()
.build();
We could even make builder_unsafe an unsafe function that you need to call inside an unsafe block, just to make sure the developer knows what he’s doing.
Granted, this is all on the Rust side, it would be a good idea to extend this to other clients that let you build and send XCMs.
Having this on the client side and something that validates fee payment on the config, or at least warns or is at least a default config that most people will pick, seems to be a huge win.
IMHO that’s a misrepresentation, the PR does change a method, but it does many more things like changes/corrects configurations for multiple runtimes and adds many (previously missing) asset transfer tests to pallets, runtimes and xcm-emulators.
And crosschain interaction is just hard with XCM.
But I do agree on the core of your message ^
There is a lot of work remaining to build/expose more builder-friendly constructs. As for end-users, I honestly don’t see an option to expose-to/expect them doing any raw XCM programs themselves (unless incredibly basic/simple).
The following is my opinion from a non-core dev perspective. But from a not-super-technical guy that has been tinkering and helping around with XCM related things for almost 2 years.
My main concern is how difficult is to use and debug XCM related stuff from the mindset of not a core developer.
Every time I hit XCM issues, the answer is “add logs and compile a custom binary”, or I have to use some scripts we built to decode the incoming XCM into a human readable form and do some nonsense debugging.
XCM is a tool for developers building on top of parachains to use. It is not a tool for core developers.
One of the recent examples of a change that just made using it much worse (from a debugging perspective), was the introduction of the messageQueue pallet and error handling. Now, when there is an issue the most common error I see is Unsupported. Now debugging issues is a nightmare as there is very little indication of what is going on.
Another crucial thing is the need for at least two runtime API calls:
One where you provide a message in bytes to the dest chain and it returns the weight (refTime and proofSize) and the fee in whichever token you are buying execution with
One where you provide the XCM message and it returns if it will be executed successfully or not, and it returns an error that is descriptive to the developer
XCM should be built as a product for dApp developers, not for core-devs. Also, having a database of real use cases, common errors, different XCM combinations, would be helpful.
Thank you everyone for providing feedback on this matter.
There’s definitely a lot to be done, but we’ll get there.
I’d say the most important things are the following:
having a better story around xcm configurations, be that defaults, composability, compile-time safety checks or warnings
having a better story around debugging, being able to figure out the exact error and troubleshoot a failed message without needing to replay the execution with logs enabled
better guides and examples on how to do simple things with xcm
I’m going to be looking into these things and keep giving updates on them. For the guides and examples, I’m planning on using the new syntax that I shared on this thread.
If you want to suggest other issues for the guides, let me know. All help is welcome.
Is the idea to have these new “standards” as part of the XCM instruction set? Because then, we depend on Parity to include this into the staging-xcm crate, right? Or are there any plans to allow to extend the instruction set easily without having the same set of instructions in every executor?
I would be interested in whether the plan is to further extend the instruction set depending on which use-cases are relevant enough?
As most standards emerge and solidify through the simple fact that they are used by the majority I feel like this will be a hard route, as it leaves little room to explore different approaches. And by the fact, that the whole ecosystem relies on their ideas being accepted by one repository.
Blockchain engineers that are building parachains with cross-chain features?
Then:
We need an XCM Playground
An interactive XCM playground leveraging the XCM emulator
Pick a parachain’s runtime and interact with it
Play with XCM config
Templates for examples
AND/OR
Dapp developers that want to build cross-chain dapps?
If so then you need to make:
XCM easy for dapp developers wanting to build cross-chain dapps
e.g. build a higher-level abstraction layer, one XCM API to interact with all parachains
Beat the competitors. Include competitor research in your product development. What are the competitors doing? How can we outperform them on every angle, including DevEx
Once accepted, a new instruction, or a set of them, is created and this widely used interaction can be done in a much simpler way
Or are there any plans to allow to extend the instruction set easily without having the same set of instructions in every executor?
No chain needs to support all of the instructions, that’s why the executor is so configurable. If your chain for example doesn’t support locking assets then it would just configure AssetLocker = ().
The instruction set growing doesn’t mean everyone has to implement it. It just means that that’s a standard interaction users could have on a consensus system.
As most standards emerge and solidify through the simple fact that they are used by the majority I feel like this will be a hard route, as it leaves little room to explore different approaches
Different approaches can still be explored via the Transact instruction. The whole idea of having a standard is that it’s a standard everyone follows, that’s why it’s one repository. Even if your chain has no useful interpretations of some instructions, we all still agree on one language and we all work towards defining it. That is how I think XCM should evolve.
I’d like to add a brief input based on our most recent Trappist experience, which is intended to be an xcm playground among other things.
Most of what we detected as pain-points during the last period was solved by the xcm-docs so that is awesome.
Also I would like to support the idea of putting extra efforts on the debugging experience and resources since we had to go through a lot of logs while implementing some use cases, which is doable but I think it scares devs out. Spoke with some parachain team members trying to implement some stuff and many of them were not really confident using them (custom logs or even current prod logs).
I also believe that the information is out there but more examples of how to use both available tools and the development process could help. By examples I mean that it would be useful to add more cases of configurations and so. I understand the concept of the flexibility and the power of it and that basically makes it really impossible to cover all applications. However, I think it would be great to add some 0 to hero tutorials of building A, B or C real world case, this probably would get developers into the mindset and allow them to fulfill their personal need much easier.
We had to set up a zombienet and played a lot with PJS which was not the fastest way to advance, I think that the Emulator es currently standing in a position where it can become the place to go for building stuff more quickly so I would also suggest further documentation and examples. Simulator could be useful for some simple stuff but it gets outdated pretty fast and I wouldn’t incentive team to maintain it.
Finally, xcm-docs does approach it but would go deeper into explaining how execution is paid or some recommended patterns from core side to approach this so we try to move into a standard proposition at least of how this could be handled, I know they should know but already seen a team getting it’s asset lost due to some missing instruction or conf.
Overall I believe that it would be beneficial to focus into showcasing and teaching to be able to move into standardization discussions since I do believe that a lot of community members understand some concepts but they really don’t know how to “use XCM”.
Hope this provides any value and happy to support on any effort related <3
You are missing out if you are not using Chopsticks to debug XCM. Compare to Chopsticks, Zombienet and Emulator are just not the right tool for this task.
I definitely agree that Chopsticks is a true blessing for dry-running and debugging XCM. I was wondering whether you could recommend any best practices for debugging @xlc. I have started using Chopsticks only recently and my debugging process has been a really basic trial-error-approach of dry-running and looking up the errors in the code. Given that the latter are sometimes quite opaque, it often feels like looking for a needle in the hay.
There are many small things could be helpful. I am probably going to give a talk on this topic in one of the future speaking chance.
A list out of my head:
Use dev_setHead to revert block. So you can send a tx to do something, see it doesn’t work, revert the block, and try something else. Useful if setup is complicated.
Override storages directly do make certain tasks easier. e.g. there is no sudo on polkadot runtime, but you can directly modify the storage of the scheduler to dispatch root call. https://twitter.com/XiliangChen/status/1721656624782434568
Make a script if you need to repeated setup the chain for testing. Manual repetitive tasks is always a productivity killer. e.g. create a script to connect to the Chopsticks RPC, use dev_setHead to set the chain to clean state, run some code to test. And you can just a single line to run the test instead of like clicking 20 buttons.