ParityTech update for April 2024

Hello all,

Our updates for April start with three issues on chains: we apologise for the inconvenience and we are working on improving how we detect incidents, manage them and get these issues fixed once and for all.

We have analysis and postmortems available:

  • Polkadot Parachains stalled on 21.04.2024 for around ~1 hour. The enabling of new subsystems on the node side, triggered by the runtime upgrade, led to parachains not being able to make progress. This issue was related to the enabling of asynchronous backing. For more information check out the postmortem.
  • The switch to Coretime on Kusama didn’t migrate the full state to the Coretime chain. The runtime that was deployed on Kusama was expected to send all the information about all parachains with a lease to the Coretime chain via XCM. While this worked, one of the XCM messages failed to execute on the Coretime chain. For more information check out the postmortem.
  • There was also a collator selection bug. Affected group was pretty small, some collators got their funds frozen until the next upgrade. Issue is fixed.

On a more positive side:

  • JAM is out of the box and you can read the FAQ and dive deeper by reading the research paper at I strongly advise you to print it and to read it a few times. If you want to diagonalize it, read sections 1,2 and 17,18.
  • You can read below about a slew of releases, a mix of large ones (bridge, async backing, coretime) and a long list of improvements. Polkadot.js, Asset Transfer API and Subxt are making regular progress and are contributing to a smoother experience on top of Polkadot.
  • Parity is excited to announce that ink! will be collectively shaped by a number of ecosystem teams going forward. The R0GUE team will initially steward the new journey, and ink! will continue to thrive with support from Aleph Zero and Scio Labs. Read more about it in this blog.

And now, a team-by-team update.

  • Product Engineering
    • Many polkadot.js libraries and tools have been improved with several features (like updated Runtime Definitions, compatibility layer for H160 addresses and upgraded to latest chopsticks). Also this incoming change should ease and improve the release process for the entire polkadot.js namespace.
    • The most important JSON-RPC APIs (chainHead for obtaining the current state of a chain, and transaction/transactionWatch for submitting transactions) have now been stabilized to v1, and the corresponding PRs in polkadot-sdk have been merged. This marks a key milestone in providing a new, well specified interface for libraries like PAPI and Subxt to build on.
    • Subxt has now exposed all of its core functionality in a new subxt-core crate which will soon be published to This crate is no-std and WASM compatible, allowing for things like encoding and signing transactions and decoding blocks, storage and events on many more targets. We’ve also added (experimental) support for singing Ethereum style transactions which is compatible with chains like Moonbeam and Mythos.
    • Substrate connect is closer to its new vision with several features added in terms of account management and keys management.
    • The new release for API Sidecar will add new endpoints for the new Runtime APIs to query the metadata and fixes for the new staking rewards storage interfaces.
    • Several features have been added to Asset Transfer API paving the way to adding complete support for bridged assets.
    • The new release for Txwrapper adds support for the asset conversion pallet and the pool assets (liquidity tokens on Asset Hub).
    • Support Parachain’s Coretime migration through a new tracking tool so teams can understand when things are migrated and what they should expect (GitHub Repo).
    • Support Snowbridge: Worked closely with the Snowfork team to create a detailed Snowbridge walkthrough, transferring tokens from Rococo to Sepoia.
    • 1-Click Deployment Product: With Zeeve’s DF grant approval, we’re working closely with them to deliver the first 1-click deployment production-ready product. An MVP of using OZ templates with POP and deployed with Zeeve can be found here.
    • OpenZeppelin: The new runtime is scoped to deliver an EVM-compatible runtime with coretime capabilities out of the box by the end of May.
    • Alpha Program: Over the month of April, 441 projects were migrated and 13 new teams joined the Alpha Program. A feedback portal with was introduced, along with the list of resources from the program.
    • Paseo: With 7 parachains onboarded so far, and successfully concluded the first quarter payment to the relevant teams.
    • Developer Hero Sunset: The Dev Heroes program has been sunset and communication with the group of developers will continue via Discord, which will be self-driven and run via the community.
    • New blog posts on Async backing focusing on the more educational side of the tech, Agile Coretime highlighting its value for Polkadot), and one on ETHDenver to share our experience and impact at the conference.
  • Chains
    • Launches
      • Asynchronous backing: Launched on Kusama, enabled runtime on Polkadot.
      • Agile Coretime: Launched on Kusama, Polkadot launch is in preparation.
    • In development:
      • Network stack rewrite has been merged. With the new polkadot-sdk release it should be possible to test litep2p using the flag --network-backend litep2p. It is not yet recommended to use this on any production chain. Tests are ongoing and fixes still being applied. The new networking rewrite is expected to bring huge improvements on the front of CPU and also finally bring WebRTC which is quite important for light clients in the browser.
      • First iteration of elastic scaling support is up-for-review introducing the slot collator, allowing a parachain to use multiple cores. It has been deployed on the versi test network where initial results were successful
      • Initial experiments to create an omninode (capable of syncing different chains) are still on-going, there is a prototype available.
      • Fork-aware transaction pool rewrite is on-going with an implementation and PR going up soon. Currently there is a bug which leads to reject transactions as invalid while they are actually valid. This only happens if the transaction is send by the same sender and is only seen on parachains. Thus, after this is fixed, parachains should have a better user experience.
  • Runtime
    • Polkadot / Kusama Runtimes
    • XCM
      • Extrinsic for trapped assets
      • Fixed major fee problem for wallets and parachains when an account drops below ED during execution
      • Progress on XCM fee estimation
    • FRAME
      • Token docs in FRAME are revamped
      • Revamp of nomination pools to make the tokens usable in governance is finally in the last phases and will be deployed on Westend soon. Follow the progress here, and shoutout to Ankan for making this happen
      • On that note, the safeguard limits on nomination pools’ for Polkadot has also been lifted via this governance ref
      • The benchmarking CLI now lives in its own binary crate and can be decoupled from the node side. The try-runtime subcommand has also been fully removed, as it is a standalone CLI now.
      • We’re experimenting with a single polkadot-sdk umbrella crate to make it easier to use our released crates without dealing with version conflicts. In all other cases, check out psvm as a handy tool
      • On a similar note, the umbrella crate for frame has been deployed as polkadot-sdk-frame and can be used as a single dependency for pallet and runtime development
      • The templates maintained by parity are being revamped as described here
  • Infra & Data
    • Infra & Data had its second quarterly planning meeting, setting the course for Q2 2024. Major objectives include “Stabilizing Polkadot” with a series of tests, monitoring and more tools to assist engineers.
    • Parity Data team proposal has passed :partying_face:: we’re extremely grateful for the trust the community has put in our team and we’re now working on the final details for the legal agreements between Parity & Token Terminal.
    • Forklift cargo caching was open sourced successfully, write-ups are being prepared to be shared with the community. This tool will allow faster build times with Rust and make building the Polkadot-SDK quite faster.
    • We’ve supported the Polkadot <> Kusama bridge launch by deploying the first Polkadot <> Kusama bridge. We’re working with the team on improving the deployments and testing around it. We’ve also deployed coretime chain collators.
    • Further deployments include (but are not limited to) the addition of WSS bootnodes for Rococo/Westend to allow light-client enabled DApps to work on testnets.
    • Data added BridgeHubs & more chains to the data ingest, many of which were direct community requests. Our goal is full coverage of Polkadot.
    • The Messari proposal passed: the data team has delivered all the data necessary for the Messari Q1 2024 report and will provide additional data for further deep dives.
    • The DotLake Twitter/X account reached 300 followers!
      • Data was picked up by the Polkadot account, Paritytech and was featured on CoinTelegraph. We’re extremely proud to “put Polkadot data on the map” and ensure proper representation of all of the ecosystem.
      • Similarly, we’re supporting and helping Distractive with some data/datasets
    • We’ve also implemented general improvements to the Continuous Integration
      • Integration of GitHub runners in Thanos (for logs)
      • First GitHub Actions job in polkadot-sdk, preparing for a bigger migration
      • Code coverage project has begun, as part of the “Stabilizing Polkadot” initiative
    • Our teams are on track to migrate faucets to PAPI, helping out the team with feedback and testing
  • Security
    • Incident Response: one external event happened at the beginning of the month with the XZ backdoor focusing on supply chain attack.
    • Code Audit: Regular audit continues to be performed internally and in partnership with audit firms.
      • OpenZeppelin’s Vanilla Template has been audited and the report will be published next month.
      • 7 disclosures of vulnerabilities have happened in April. And a new page will be made available next month to help people navigate the different disclosure done and associated status and guidance.
      • 3 Alpha Program projects have been selected for a Security review in Q1 and the same approach is being done in Q2.
    • Parity’s Security Bug Bounty: 9 submissions have been done, none critical.
    • Polkadot <> Kusama Bridge Security Bug Bounty has been approved thanks to the support of the community. A group of 7 curators from the ecosystem will support and evaluate the effectiveness of the bounty and the attribution of the reward to the reporter. For the top 5 people on the Bounty Hall of Fame, if they wish their candidacy to the Polkadot Blockchain Academy will be considered in priority and the Top1 will have a slot reserved.

Threat Intelligence: As presented at Sub0, partnership is progressing with the SecurityAlliance(SEAL) in order to improve information sharing between ecosystems and be more proactive than bad guys. It is free to access.


any tentative timeline for Nomination pools: Allow funds to be used for democracy ?

we are getting frequent queries for this at the Pokadot discord.


@muddlebee Always tricky to give a timeline :slightly_smiling_face:. The feature is code complete, has got at least one approval and is pretty imminent on Westend. It is still undergoing audit, and if no new issues/bugs are found, could potentially go live on Kusama/Polkadot by early Q3.


Good to hear about this

The provided solution fixes an error introduced in SDK v1.7.0 but was not backported and is only part of SDK v1.11.0. Hereby, I want to express my dissatisfaction with that.

At Centrifuge, we are currently updating our Codebase from SDK v1.1.0 to v1.7.2. Thanks to Chopsticks, I noticed a reduced list of queued session keys as well as an empty collator candidate list when applying our WASM to the live state. Seeing that we were in sync with the latest storage version v1 of the collator-selection pallet, I assumed the issue was introduced on our side. Only after deeper digging, I noticed the breaking change in the mentioned pallet without any fix for our dependency version.

It is really frustrating as a builder to have to deal with such bugs introduced from the SDK even though there is already a fix available. Another example is the parachain_system::HostConfiguration breaking storage change without bumping the storage version or providing a migration. It appears to me that the SDK needs to have stricter checks for any storage changes!

1 Like

As I already told you in the issue, we have these checks now. This breaking change was introduced last year in August and since then, we have increased the number of tests. Especially when it comes to ensuring that storage items do not silently change. The try-runtime checks for example now decode the entire state, which helps detecting these kind of bugs. Especially when it comes to this bug you mentioned, I should have requested that we bring up a fix. Sorry for that!

I also get this and we are working hard to bring up even more tests that this is not happening again. The entire release situation should hopefully greatly improve with the introduction of the new release process. Especially as there will be less releases and we can focus better on backporting, if required. I can coordinate internally that this fix gets backported.

1 Like

Sorry for re-rubbing salt in the wound, I just wanted to mention this to proof my point.

Despite the frustrating situation, I really appreciate the lightning fast speed of your responses @bkchr! It is incomprehensible to me how you can be everywhere so fast and on point. Have you succeeded in cloning yourself? :sweat_smile:

I would really appreciate this and I believe other parachain builders would as well, especially because it decreases likelihood of parachains missing the migration and reducing their block authors to the invulnerable set. Of course, manually backporting it wasn’t an issue but this bumps the collator-selection storage version (v2) above it’s maximum (v1) until Polkadot SDK v1.11.0 which leads to unsuccessful try-runtime executions until updating to v1.11.0:

[2024-06-06T09:06:24Z ERROR runtime::frame-support] CollatorSelection: On chain storage version StorageVersion(2) doesn't match current storage version StorageVersion(1).
1 Like

I get the frustration :see_no_evil: I hope this will be gone in the future!

I can not comment on this topic :wink:

It was actually backported to 1.7.0: Backport 4229 Into v1.7.0 by joepetrowski · Pull Request #4288 · paritytech/polkadot-sdk · GitHub

However, it was only backported to the crates release branch (another thing that should not exist with the new release process!). I assume you used directly the release-polkadot-v1.7.0 branch? I will push the backport there, so you don’t require your manual backport.

1 Like

Yes, exactly. We hope to switch to crate versions soon but for now, we prefer the release branch approach because it is easier to sync between multiple repositories.

Hi, we actually use v1.7.2. We pointed that branch expected latest fixes/changes. But, should we better use 1.7.0 instead?

It seems like other features are being backported to 1.7.0 but not to 1.7.2

On request of @WilliamFreude via DM, I have backported the CheckMetadataHash signed extension and the fix for the collator selection pallet to release-polkadot-v1.7.2.