The last runtime upgrade of Statemine increases tx fee by 10x, and it breaks XCM transfer from Karura to Statemine due to the configured fee is no longer enough to buy execution and ends up assets trapped.
Karura have now updated the tx fee config and should be able to claim the trapped assets back to rescue the funds in a future governance proposal.
This is not the first time Statemine upgrade breaks parachain integration. And we know there will be a big breaking change coming later this year: XCM V3. No doubt one chain will upgrade before others and there is a good chance we have some compatibility issue during the upgrade. Therefore we need to have a solution to test XCM compatibility between runtime versions.
Shaun from our team have developed XCM emulator (in addition to XCM simulator, which is contributed into Polkadot now) with the goal to be able to write unit tests to test XCM related code and that works fine. However, we can’t use this to test compatibility between versions. This is because at this stage, it is not possible to compile two runtimes using different version of Substrate in a single Rust binary. So we can’t test Statemine with XCM v3 with old Karura with XCM v2. We need a different solution.
Chopsticks have recently implemented the ability to launch multiple networks and transport XCM between them. Combined with the ability to override Wasm and perform runtime upgrade, we should be able to create a tool to leverage Chopsticks for e2e tests and ensure future runtime upgrade is compatible with other networks.
However, this must be a mutual effort between all the parachains connected with XCM to ensure the tests are up-to-date and run the tests with new runtime and notify others on breaking changes.
I have some vague idea on how this could look like but I am refraining propose it until it is getting more clear and meanwhile see what’s everyone’s opinion about this.
Please comment to signal if you are willing to commit to this effort to maintain compatibility between your parachain and all other connected parachains.
It’s been almost a kind of a solo effort, and unfortunately lately I didn’t find as much time as I’d like to keep improving the tool. I invite the community and Parity to participate on it as I strongly believe it can be very useful and can help many Parachains teams to test interactions with each other.
Hi!
From Mangata we are creating some e2e with chopsticks tools too. Those tests will help us on:
Validating with the current status of any chain ( Kusama for us )
Test that our integration works
Test that our code under test overwritten by the wasm does not break any compatibility.
I also personally considered setup a heartbeat to receive some alerts when they fail.
Since it’s an effort from both sides ( parachainA → integrates with → parachainB ) I consider that a mutual effort to keep this compatibility too.
Unless we start enforcing parachain to have an environment to test xcm keeping the latest status of our code before releasing as a barrier, this issue will always pop up.
Our tests are running against latest block (and therefore a bit flaky) and we plan to make it run nightly so it can notify breaking changes on mainnet.
Since it’s an effort from both sides ( parachainA → integrates with → parachainB ) I consider that a mutual effort to keep this compatibility too.
100%. That’s why I want to create a common replace for all those tests to reduce duplicated work.
In a few words I would propose:
– On rococo ( and perhaps later on Kusama / Polkadot ),
Have a status check UI where all the para-chain devs add their para-chain and configure the different channels / assets / weights, then we run a chopstick and trigger all the configured xcms back and forth. As result it will be a fail / pass so perhaps sending a notification / update the status on the UI would be a nice status report.
The workflow would be something like:
Two parachains are connected
Those two parachains are configured correctly in our platform.
One of the parachains adds a bracking change on their last runtime upgrade
One of the tests that are triggered each-n hours fails, so a notification is sent to both parachain Owners.
Common agreement, parachain configs are fixed or version rolled back.
Evth is green again
This looks to me like a parachain-matrix-healtcheck tool
Why rococo if we can just do it with Kusama / Polkadot?
Because I think everybody is interested on detecting incompatibility issues before releasing.
Also what you are proposing is basically e2e tests abstracted into config files.
Kind of (could be implemented by that), but I think that the main point is that it should be easy for everybody to add their config easily so the test are usable for everybody. And about reporting, would be awesome have a single page to check the status of the different para-chain status.
Because I think everybody is interested on detecting incompatibility issues before releasing.
That’s exactly the reason we want to do it on (fork of) Kusama / Polkadot. Rococo is way too different.
Kind of (could be implemented by that), but I think that the main point is that it should be easy for everybody to add their config easily so the test are usable for everybody. And about reporting, would be awesome have a single page to check the status of the different para-chain status.
I believe it is now on a good shape that should be relatively easy to add new chain and new tests. I would like to invite everyone to take a look and help contribute XCM tests for your chain. Also I would like to ask parachain teams to use this to verify compatibility or your next runtime. If you don’t expect breaking change, you can just set environment variable {CHAIN}_WASM with path to new wasm runtime and you can run all the XCM tests with your new runtime to ensure it does not contain unexpected breakage.
Sure. I am still trying to figure out the best way to structure the code to balance between generic and chain specific. Have more chains will help to catch incompatibility if there are any.
