How XCM will actually be used with XCMP

NOTE: This was a thread which took place in the internal Parity Forum on July 12, 2022, and would like to now migrate to this public forum now that it exists. I will try to copy and paste relevant parts of the conversation, and would love for us to continue the conversation here.

In a conversation here at the Polkadot Blockchain Academy in Cambridge it became more clear to me how to describe how XCMP / Polkadot can be used effectively versus some of the misunderstandings other have about it in our ecosystem.

I am of course not an expert on all things here, so maybe some of this is wrong, would love to hear from experts like @rob and @gav

Polkadot’s bet is quite simple:

  • Scale by parallelizing transactions to multiple blockchains. (sharding)
  • Scale by having those multiple blockchains be specialized and optimized for a particular use case. (heterogeneous)

However, these choices also come at costs:

  • Technical overhead and complexity of building these systems safely and securely. Mostly on Polkadot protocol development side, presumably why something like this takes 5+ years to develop, and many more to come. Something we can overcome.
  • Sending messages across blockchain shards is slow. Like WAY slower than you can expect for any “cross contract” communication that people have become use to with defi and all that. Something which I argue cannot be solved.

It seems to me that people want to duplicate multi-contract scenarios with parachains, but if we do this, Polkadot will look bad, because Polkadot (or anyone else) simply cannot do this. It is literally beyond what physics allows.

Polkadot instead makes the bet that a user will not actually move assets across Parachains that much, and when they do, the advantages of a specialized chain will be worth it.

Let’s imagine two dapps: a DEX and and a privacy layer like Tornado cash.
Let’s assume a DEX contract on Ethereum can perform a swap in .2 seconds.
While a dedicated DEX parachain can perform a swap in .1 seconds.
Let’s assume a tornado cash contract can perform a privacy operation in 1 second.
While a dedicated privacy parachain can perform the same operation in .5 seconds.

NOTE: I think this is a vast underestimate. I think there is evidence that parachains are hundreds or thousands of times faster than their contract counterparts.

Now let’s assume that on Ethereum, moving a dex swap token into a privacy layer is basically 0 overhead.
While on Polkadot, we should expect at least 12 seconds for an XCMP transfer between the two parachains.

This means, if we are trying to highlight scenarios where a user can quickly transfer some token to a dex, swap some token, then move it to a privacy layer to hide their trades, and then back to some final account, we will be so far behind Ethereum it is not even reasonable.

Instead, the scenario which highlights Polkadot’s advantage is when you are doing 10,000 dex swaps, then a single transfer to a privacy layer to hide all those operations. Here, you would actually save much more time, because each of those dex operations have been optimized on a dedicated blockchain, and the overhead of XCMP is shadowed by all of the time (and fees) saved by doing it in a more optimized environment.

What does this imply about how users and products will be built on Polkadot?

Well, Parachains won’t actually be THAT focused, at least, they will all likely provide “good enough” solutions to common operations like DEX (if they manage multiple tokens), smart contracts to a certain extent (for extending funtionality on their chain), etc…

These parachains will have these applications as secondary features compared to their main product. And they can achieve these features quite simply thanks to Substrate and our modular pallet system. Adding a dex to any parachain will be as simple as adding a new pallet to their runtime.

However, this does not mean the death of DEX parachains, since we all know that systems like a DEX are optimized where there is high liquidity, lots of different available assets, and only a dedicated DEX chain will be able to provide that as a primary feature. If you are doing a small conversion for some scenario, probably you don’t mind too much about a super optimized dex trade, but if you are doing big trades, probably you will want to eat the time cost of transferring the asset to an appropriately competitive DEX parachain. Same with contract parachains.

This also implies that users will probably end up having a “usable” amount of assets on each chain they care about. It will not be a normal scenario that a user wants to perform one action on a parachain, so they start with their tokens on DOT, initiate a transfer with “just enough”, and perform that one operation. Instead, they will probably predict how much they will use another parachain over a long period of time, transfer enough assets to that chain to handle all of those operations (and more), and just manage that balance on the other parachain.

This means we probably should be directing teams to build less UIs which execute operations across multiple parachains, and instead focus on dedicated UIs for all the operations on a single parachain, and a really amazing UI for managing your assets across multiple chains, and making that one time transfer.

Does this mean that all of the programmability of XCM is useless? No, since XCM is also targeting contracts where all of these cross contract calls are free (our initial assumption), but likely XCMP and cross parachain programmability will be minimal, since the amount of “actions” you can take on either side of the XCMP operation won’t be that large, and thus does not really outweigh the large overhead of the transfer itself.

There are probably a lot more that we can derive about behaviors and what will actually be successful on Polkadot if we continue to think down this path.

Is there anything here which I am getting wrong?

Are there any bets we are making which would change if this is right?

4 Likes

Here is a collection of responses from the thread:

  1. Gav

    I suspect that it won’t be applications which parachains rally around and which defines the boundary points, but rather application-classes (what might be termed “on-chain industries”), and the users and communities which come with them.

    So it’s not one parachain for privacy and one for a DEX, but rather several for defi each with their own marginal advantages (localities, community, features, stability/innovation tradeoff, …).

  2. @santi

    This is indeed a very interesting discussion. Interoperability has been always a key aspect on the way we advertise Polkadot and understanding what this really means is indeed important.

    I can see how a DEX could exist in multiple parachains as a secondary product (maybe even the users don’t know it exists), although I wonder if there could be some potential liquidity issues. Even how DeFi projects could be more targeted one way or another but eventually offer similar products.

    However, there might be other verticals that can bring solutions that work across all these projects, and that by composing with other parachains is that really bring value. How will a an Identity parachain then work if not by heavily interoperating with other parachains in the ecosystem? Practically speaking, I can imagine that I don’t only want to have a verified identity for the sake of it, or that I need to verify multiple identities together, but that I would want to have a verified identity to be able to do things somewhere else in the ecosystem. For example, it might be the case that a Loan specific parachain tells their users that if they verify their identity, they could get extra benefits. Same thing in my mind goes for a Files Storage parachain. Could a potential File Storage parachain serve as a base service in the ecosystem that other parachains use and fire actions towards to save, edit, change ownership or any other operation of some metadata? These might be examples where functionality triumphs over speed and can be treated differently. Could these be the common-good chains altogether?

    Last DQ I have on my end:

    This also implies that users will probably end up having a “usable” amount of assets on each chain they care about.

    How do we see Statemint working in this scenario?

  3. @natalie

    My current thinking (but happy to be challenged) is that the tradeoffs
    a) aren’t too bad (will be mitigated by how multi-chain apps are built) and
    b) don’t come unexpected, esp. when talking about more than asset transfers (and you’re right that we should be careful in our communications, especially down the funnel)

    I think the more specialised the features are that are utilised by other chains, the happier users are to wait. Dapps will be built around these limitations, balancing the UX (wait time for the user) and effort (time to build & maintain a “good enough version”).

    Similar to integrations in Web2. If I want to pay with my credit card within a new product, I’m happy to wait if a VisaSecure pop-up needs a few minutes. The users aren’t upset because they know that the whole process of contacting Visa for KYC, 2FA, vendor / payor verification, settlement, is complex (or I just bear with it because it only happens once). If I do frequent transfers in an app, I expect other solutions, my card to be pre-authorised, etc.

    → yes, would love to think more about the on-chain industries

    → Not sure how much we guide the parachains, but it would sound reasonable to brief them and collect some best-practices along those lines

    So all in all, I don’t think this contradicts our story of the multi-chain future.

    Hope that made sense and happy to hear your thoughts!

  4. @shawntabrizi

    I don’t think we should really guide teams directly, but since this take does seem pretty accurate, it would be good to have it shared with these teams so they can consider these situations themselves.

    I think having teams transition to thinking about themselves as “on-chain industries”, can be healthy to the overall narrative, and help teams really understand how to build successful products on the platform.

    If it is good written as is, perhaps it can be copied to one of the appropriate parachain chats? Or i guess we can make a short presentation and get a video out there?

  5. Travis Kassab

    Yeah, those conversations are happening between parachains on separate channels. Which I think is fine they seek out specific synergies themselves.

    It seems parachains will increasingly want technical validation from Parity once they have a concrete use-case. As in, “is this feasible? is this sensible? given the technology”

    To be more proactive, Parity could put together a list of principles: “Cross-Chain Design Considerations with XCM”.

    @shawntabrizi started this by comparing multi-contract vs cross-chain. Maybe it makes sense to boil down blockchain UX (e.g. time to finality, fees, liquidity, risk/security) and compare other scenarios.

    ^I’m sure this all can be implied from technical documentation, but spelling it out for parachain PM’s could be helpful.

    To me, the best Web3 UX I’ve seen have been aggregators. To tag onto your DEX example… DEX aggregators are great because they get to the heart of issue: the user wants a token as quickly, and as cheaply (less fee, less slippage), as possible. The user does not care which protocol executes the swap.

    This may be more long-term, but I think we should be framing the UI conversation around cross-chain aggregators that service an on-chain industry.

    You can imagine within a DeFi UI, would be a cross-chain DEX that optimizes time to finality, fees, and slippage with either:

    1. local swap using parachain DEX
    2. transfer assets to other parachain, then make swap
    3. bridge assets to other ecosystem (e.g. Ethereum), then make swap

    Although, I don’t think parachains are indentivized to build these aggregators because, in some cases, it would drive liquidity off their chain. Still, I think aggregators that utilize multiple protocols on their backend, and obfuscate this complexity, will win out.

NOTE: some comments skipped here…

  1. Gav

    I’m still not entirely certain what the point of these “design guidelines” are. Cross-chain use-cases should come about naturally by teams seeing an on-chain service offered and having their own on-chain service utilise it to create a new (composite) service.

    This is generally possible to do without using XCM in an especially sophisticated manner - Transact is all that is really needed at the core. That said, there are good reasons why we might want to evolve XCM to deliver a more ergonomic cross-chain language for doing certain useful operations (as is already the case with asset ownership manipulation).

    The main points to always remember when thinking about XCM is that it’s a language designed primarily around the communication of remote systems (parachains/XCMP being the primary expected case): It is asynchronous and it doesn’t provide any first-class return path for results, essentially giving the same interface as transactions. There is little language support in Frame for acting on results between blocks (i.e. futures/green threads) and it is optimised around the assumption that transport layers are not instant.

    This all means that use-cases requiring high degrees of coherence are probably not going to work well across XCMP/XCM for the moment.

  2. Gav

    You can think of XCM, at its very basic level, as allowing one blockchain system to post a transaction to another blockchain system. This means it’s a facility for a blockchain, not a user. Use-cases are limited to the blockchain team/developer, not the public at large. The prototypical use-case is to allow a blockchain developer to have their chain rely on a service offered by another chain, rather then deploying some similar service and competing over users. The limitations are exactly the same as transacting on a blockchain: there’s a delay, it’s simplex and asynchronous.


There are a few more posts after this, but will let the conversation restart here.

Reposting from Internal Parity Forum:

I feel like such a conclusion is narrowly focused on the use case of DeFi or DEXes. It’s normal for these businesses to be time-sensitive, but for other kinds of usages where waiting for execution isn’t really a big deal such as identity verification (like Kilt or Litentry), then I don’t really see why they can’t continue to focus on being a parachain that focuses on a single use-case. I don’t foresee them ever adding an on-chain DEX, if that’s what the implications are from this thread.

But I do see the need for other DeFi or DEX chains adding on an on-chain identity verification pallet, should they want to verify the identity of the origin for every transaction they make. It would be too costly in terms of time to use XCM and communicate with an DID parachain to get an individual verified for each tx. This would sound to me that for applications that are time-critical, they’ll really be deploying their own versions of various services on their own chain, and if this is the direction that we’re heading towards, then it would make sense to encourage non-time-critical applications to provide a pallet-as-a-service to time-critical parachains.

2 Likes

The above two quotes really stood out for me. There has been discussion among some of the polkadot communities I am a part of that we are at risk of trying to make polkadot do what Ethereum does. Given that is where the majority of traffic is I understand the decision but as @shawntabrizi highlighted above, Polkadot can’t and should not compete there. Those same users are arguably not our key demographic given how they use it.

The key selling point of Polkadot for us was interoperability, XCM is core to that. However, what needs to be highlighted is how it is currently being used as a token transfer mechanism which while works is really just a low-hanging fruit

IMHO, The true value of XCM is in the communication between chains, that communication is what drives the innovation. Similar to how the smartphone boom happened, now your phone could be much more than a phone. It had apps, maps, and social media to name but a few. All made possible because your phone can communicate with other services. We need to look at that as the next frontier to drive the industries that Gav mentioned above

Also, I agree that Transact could do that, we have done it before where we had the same pallet in 2 chains and used Transact to call functions in other chains but that was a rudimentary hack to work around the fact that XCM was one way and did not provide a response. That said, I think Transact will need to be worked on a bit to make it more intuitive for a blockchain developer. Documentation and education will also help. Make devs realise it’s potential

I also agree there are a lot of teams waiting for Parity to define what “good” looks like, how they communicate or a best practice. I would go as far as to say Parity themselves do not know, how could they when it has not been done before? Honestly, I think it is unreasonable to expect them to define it either given each chain is focused on it’s own business use case

My question is why not focus on communication, abstract away “how” chains communicate and just build demos and proof of concepts on what could be done with the communication? show the end user where the true value is even if we simulate the communication while we get more devs to understand how to utilise Transact?

We don’t need Parity to do this. We - the builders - can.

This gives us invaluable feedback as to how people will want to utilise this communication as well as give us an idea to what an ideal set-up should look like and give Parity ideas on how to improve on Transact

I would also be remiss to say that we need to move away from the DEFI narrative and think about who our target users are. If we are to push for that next wave of adoption of users then I am not certain DEFI is what they want to use on a day-to-day bases. DEFI should be a means to an end, not an end itself.

In summary, the post questions how XCM will be used with XCMP?

I would go further and ask how can the value of communication between chains be demonstrated conclusively. Not in theory but also in practice

6 Likes

I fully agree with the point made here. And I also think, like it has been said in the thread, that the more specialized a chain is on some task the more users will be willing to wait.

I think this is key. I dont think it would be very smart to transfer just enough everytime we want to take perform an action on a different chain. This makes even less sense when we think about how users will be able to do cross-chain interactions. We are starting to experiment with the scenario in which the parachain A allows an account on parachain A to control one and only one specific account on parachain B (through the DescendOrigin instruction + an address defrivation). In this scenario, this derived account on parachain B is sort of an extension (only xcm-controllable) of the account on parachain A, something that the user will use everytime it has to interact with parachain B. Thus, it is completely logical to fund this address for operations that will be performed over a large period of time.

The same thing applies with other use-cases. If we have a cross-chain liquid-staking scenario, it is more efficient to just stake once every session than everytime that the SC receives liquidity. We have a use-case around this (Lido with DOT), and this is the approach taken.

However, how do we deter the usage of cross-chain messages that go against this philosophy? This might organically happen because of two reasons:

  • Fees: Sending cross-chain messages incurs on higher fees, therefore one would expect that the use-cases that arise take this into account.

  • Byte/Weight limit: At least in the case of the Polkadot ecosystem, a cross-chain interaction is capped by size (defined by the HRMP channel config) and weight (defined by the target parachain). The more cross-chain actions taken, the more difficult it will be to fight for this space and the more a user/SC will have to wait for its cross-chain action to complete.

Because of these two reasons I think usecases will organically adapt to the philosophy of “just perform a cross-chain action when you trully need it”.

I agree with this. I wrote it before in [Discussion/Idea] Domain/Pallet specific XCM language ¡ Issue #30 ¡ paritytech/xcm-format ¡ GitHub (for different reasons), that we are not in a spot right now where we can define XCM properly.

I believe there will be more to XCM but we also have to be realistic. Until there is a technological breakthrough, the majority of activity will be around Token/NFT transfers. New scenarios are coming but those are very limited (by the nature of blockchains)

Interacting from a blockchain with the real world (which is what people want to at the end, specially if we want to do something else than Defi) requires to either trust the interaction (like our current society works) or use a physically/mathematically proven interaction (which we don’t have invented yet).
Leaning into the “trust” scenario can work (Ex: people trust tokens have real $ value) and might help to bring some new scenarios, but it is also not the direction of the promises of the blockchain technology.

Working on XCM and cross-chain in general is about preparing the base/platform for that evolution, but we don’t know what it will look like, so I believe XCM can’t be designed for it now.

However, we should still consider XCM, to help the ecosystem pave the road and learn from the challenges associated with cross-chain, in order maybe to evolve to another version or another language that will match the real scenarios we will have.

My 2 cents,
Alan

1 Like

I think this is a very interesting thread. I wrote an article a while back regarding how XCMP could be used for parachains in the future. You can check it out here. In this article, some examples I theorized might not be feasible, but overall, I think it still stands for most cases. Especially the limitations and considerations section.

Overall, I agree with the point that if the end value is greater than the added overhead that is involved with XCMP, then this should not be a problem. I see that thinking in the way of ‘industry-specific’ blockchains is more realistic than ‘application-specific’ blockchains for XCMP use cases.
So during the early phase of XCMP dApps, I imagine that most blockchains will be some sort of asset repositories but focused on assets for games, real-life assets, bridged assets, tokenized entities, automation parachains, etc. XCMP will revolve around mixing and matching these aspects with each other. This alone could be quite big.
However, I personally do not think this is a long-term application that we are looking for in a truly interoperable Web3 world for mainly the fact that industry-specific blockchains have to do more with marketing than features. You can use the same smart contract platform and claim it’s for gaming or real-life assets, and it will still work well if your initial users are the right target. Of course, you can add custom features that help adoption in certain industries, but this isn’t selling a ‘feature’ as it’s more selling your ecosystem (which is all blockchains in a nutshell IMO).
From my observations, public blockchains heavily rely on network effect and existing ecosystem, rather than new features (just look at Ethereum). Even if the value proposition and features of a blockchain is superior, if they don’t have a vast ecosystem with a lot of liquidity, most new projects won’t jump into the new blockchain. And considering that there are only a handful of industries right now that justifies a blockchain, I can easily imagine a future where only a handful of successful industry-specific blockchains have monopoly on the industry they target. Another case would be where parachains are relying on only a single parachain for specific assets or features, making a security dependency.

Now having said that, this is part of the game, and I don’t think changing how XCMP works will fix anything. It’s more in the fact that the blockchain space is quite early, we don’t have a lot of use cases. I would go far as to say that we haven’t seen any major problems that interoperability solves (since we don’t have enough cases in general).

I guess what I want to say is, we have a great protocol, but we also agree that trying to recreate the same thing that is available on a layer 1 blockchain, but spread out into multiple blockchains via XCMP is not reasonable. So this might be a good opportunity to think about what interoperability means, and what problems each parachains are trying to solve. I see XCMP as a way to unify ecosystems, not solutions.

I’m open for discussions, I can’t say my view is entirely correct here :slight_smile:

I urge you to reconsider - XCM is what brings many of us to the Polkadot ecosystem. Deterring usage of XCM would deter users and developers from joining and building in Polkadot.

As Shawn highlighted, the main problem that interoperability solves is scalability. Interoperability (and thereby XCM) enables vertical scaling by removing the need for every chain to support every use case. We’ve seen this work time and time again in software development and with Web N+1.

Gavin seemingly agrees with Sam (as do I) on XCM being much more important than simply bridging tokens. Aligning on a solution to post transactions to another chain using XCM is undoubtedly imperative to the continued growth and long-term success of the Polkadot ecosystem.

Given Gavin’s stated opinion that no clean solution exists to provide a generalized “custom proxy” solution [1], my conclusion is to agree with Sam - each parachain team must accept responsibility for their own proxy permissions.

[1] Adding Proxy Call Filters to other Time Delay Proxy Types ¡ Issue #1823 ¡ paritytech/polkadot ¡ GitHub

1 Like

This was more of a rhetorical question. There is a much more important conclusion I throw at the end:

Because of these two reasons I think usecases will organically adapt to the philosophy of “just perform a cross-chain action when you trully need it”.

2 Likes

@shawntabrizi Here it’s worth looking into the latency solutions used by the traditional financial system and perhaps adapting them to our purposes. I agree that we run into physics constraints when low latency absolute finality is desired, but there’s still room to reduce latency if we substitute insured finality for absolute.
I envision a smart contract layer that mathematically models the risk of a particular asset transfer or cross chain procedure call. The contract then adds a small fee to the transaction in return for guaranteeing its successful completion. This could be combined with batching of transactions where relevant to make things even more efficient.
Which limitations of xcmp latency could such a system help with? Intuitively it seems to me that asset transfers would be well served by such a system, while procedure calls might be much tougher to insure.

When it comes to latency, we are to some extent bounded by the blockchain scaling trilemma: of security, throughput, and latency you cannot choose all 3. There are certainly ways of bypassing XCM latency via optimistic market methods: i.e. if an XCM routine coordinating many chains might take 30 seconds to complete, it is possible for entities which know that the result will eventually resolve to X to ‘frontrun’ that message and collect a small profit for frontrunning it. The service they provide is to make information or value available before it is confirmed by the consensus system. This is a way for the application layer to selectively trade off security for latency.

So the security tradeoff in this case might be as follows? Many instances of frontrunning fail in a tight window of time. This triggers payouts from the frontrunning party to other effected parties which cumulatively bankrupt the frontrunning account, leading to unhandled damages and more importantly unsent messages.

I think about the latency in the traditional financial services system as evidence that instant finality isn’t necessary for many use cases. I’ve frequently heard the argument from banks (in the US) that much of the latency is desired (for fraud prevention).

I think this is effectively Visa / Mastercard, who process transactions instantly while waiting for responses from multiple layers of banks. They charge ~3% of most transactions. :face_with_spiral_eyes:

1 Like

At present, we are exploring the ability to provide cross-chain liquidity for other parachains based on XCM. By encapsulating the XCM call to Bifrost as a contract SDK, the contract application in the smart contract parachain can complete the remote liquidation of an asset by calling the contract (swap token), obtain the result of XCM asynchronous execution through queues and off-chain workers, and we will gain the trust of the opponent’s parachain through Bifrost Runtime version locking and verification. We don’t think this is the best use case based on XCM, but this is the direction we are trying to go, any issues with the idea are welcome to challenge (let us know, so that the cost of trial and error is reduced)

Good point. Though the existence of even a minority of use cases requiring instant finality would likely be enough to motivate latency workarounds.

Two systems can share some implementation details without sharing everything. As far as I can see the ~3% fee has more to do with a payment provider duopoly than the underlying tech.

Your thought around the multiple layers of banks is more interesting to me though. I could imagine a state where the failure of one virtual finality system, through draining of a smart contract’s token pool for instance, leads to cascading failures across many such virtual finality systems spanning many chains. That might be such a risk as to motivate us to look elsewhere.