Polkadot People Chain Stall & Degradation - postmortem

Tech Talk
1

Post-Mortem: People Polkadot Degraded Blocktime after 2.2.1 upgrade

Incident Date: 22 Apr 2026 Author: Sebastian Kunert

Summary

People Polkadot suffered from a short stall and degraded blocktimes after the runtime enabling elastic scaling (PR #1116) was enacted. The changes were included in the 2.2.1 runtime upgrade which was enacted on 22nd April 2026, 19:45. Shortly after enactment, polkadot-people stalled. The elastic-scaling related changes require the setting of additional CLI flags. After they were set, block production continued with a degraded blocktime.

The reason for the degraded blocktime is a misconfigured runtime. Polkadot-people was not correctly configured to support multiple blocks per author. Either one alone is sufficient to break elastic scaling; in practice the aura check fires first and the block-building task aborts with an error message.

Impact

  • Parachain: People Polkadot block production halted for 23 minutes. Block production is slowed down from the intended 2s per Block to 12s per Block.

  • Relay chain: No impact.

  • Users: Users of polkadot-people were unable to use the chain during the halt.

Timeline (UTC)

  • 2026-04-03 10:12: PR #1116 merged — “Enable elastic scaling with 2s block times and 3 cores” on People Polkadot.

  • 2026-04-16: Runtime 2.2.1 released.

  • 2026-04-22 19:45: Runtime upgrade enacted on People Polkadot. Chain stalls at block #4421948

  • 2026-04-22 19:52: Automated alert posts in dedicated Mainnet Alert channel in Element.

  • 2026-04-22 20:00: System chain operators report chain stall.

  • 2026-04-22 20:08: Parity engineers identify missing --authoring=slot-based as root cause for the stall.

  • 2026-04-22 20:09: Blocks produced again as collators introduce the CLI flag. First block is #4421949.

  • 2026-04-22 21:36: System chain operators report “Slot must increase” error. Blocktime is still at 12s instead of expected 2s.

  • 2026-04-23 09:19: Problem identified and PR #1154 open.

Root Cause Analysis

With BLOCK_PROCESSING_VELOCITY = 3 and SLOT_DURATION = 12000, the slot-based collator produces up to six parachain blocks within a single 12-second Aura slot. The BLOCK_PROCESSING_VELOCITY determines how many blocks are allowed per relay chain slot of 6s. The SLOT_DURATION determines how often authors rotate. Before the upgrade collators where expected to produce one block every 12s. Two runtime configurations and a missing node CLI flag stopped this from working:

  1. Runtime - Multiple Blocks Per Slot: Without the configuration to allow multiple blocks per slot set, the runtime enforces that the current slot must be strictly lower than the next slot. The second block authored within the same Aura slot hits the assertion and panics.

  2. Runtime - Minimum Period: With a MinimumPeriod of 3s, rejects timestamps that are closer to the previous one than MinimumPeriod. With a 2s block time, consecutive timestamps are ~2000 ms apart, which would trigger TooEarlyToSetTimestamp. This second failure mode is masked by the aura panic, which fires first, but would surface if only the aura toggle was fixed.

  3. Node - missing flag: Without --authoring=slot-based, nodes use the legacy lookahead collator implementation. The runtime upgrade included a relay parent offset, a mechanism to reduce the amount of forks on the chain. This feature is only supported on the newer slot-based collator implementation.

Why it wasn’t caught

  • No end-to-end tests: We have no end-to-end tests in the runtimes repo. Tests covering block authoring are present in polkadot-sdk, but not runtime specific. Runtime upgrades requiring new offchain configurations like CLI flags are hard to detect and require manual intervention.

  • Asset Hub Polkadot and Asset Hub Kusama already had the correct MinimumPeriod = ConstU64<0> and AllowMultipleBlocksPerSlot = ConstBool<true> when they enabled elastic scaling. Cross checking with these did not help in this case.

  • No enactment time: Some team members were aware of the required CLI change, but were unaware of the fast passing of the referendum. The runtime was enacted earlier than expected.

Resolution

The resolution comes in the form of configuration adjustment for polkadot-people in PR #1154. It sets AllowMultipleBlocksPerSlot = true and MinimumPeriod = 0. This allows collators to build multiple blocks in their 12s slot and in rapid succession.

Action items

  • Add an integration test on runtimes: A zombienet test that runs a 3-core collator setup would have caught both issues.

  • Always add an enactment date: All runtime upgrades should have an enactment date to make it more obvious when manual action needs to be taken. In addition, this avoids problems with nighttime deployments.

2

The tests are not in the runtimes repo, but there are many end-to-end tests in GitHub - open-web3-stack/polkadot-ecosystem-tests: Polkadot Ecosystem Tests powered by Chopsticks · GitHub . Is this something we can add tests for there (plus more diligence around running them before upgrades go live)?