[2026-06-30] Staking Election Stall PostMortem

Tech Talk
1

Summary

On 2026-06-30 the validator election on Polkadot Asset Hub (pallet-election-provider-multi-block, EPMB) stalled at round 238, the election for era 2220. Every submitted solution, signed or unsigned, was rejected with ScoreTooLow and the submitter’s deposit slashed. No solution could be queued, so the validator set, era progression, staking rewards and unbonding all froze at the last successfully-elected era, 2219.

The trigger was Referendum 1909. It lowered ChillThreshold to 32%, which opened permissionless chill_other against validators below the 10,000 DOT minimum self-stake set earlier by Ref 1890. The resulting chilling wave on 2026-06-29 removed 67 well-nominated but low-self-bond (< 10k DOT) validators, and that dropped the best achievable election score below a static MinimumScore floor that Ref 1909 did not adjust. From that point the floor rejected every solution.

The fix was Referendum 1914, lowering MinimumScore.minimal_stake from 789,555 to 500,000 DOT. It was fast-tracked on the Fellowship Whitelisted-Caller track: submitted 2026-06-30 15:35 CET, enacted 2026-07-03 07:01 CET, election unstuck at 07:10 CET (round 238 → 239). The first era rotation (2219 → 2220) followed at 13:36 CET the same day, and the normal ~24h cadence has held since (eras 2220 and 2221).

The chain itself was never at risk: block production, finality and all non-staking functionality continued normally. No user funds were at risk or lost, and no emission was lost to the stall. Unbonding was delayed, not forfeited, and resumed once the era rotated. The only amounts actually debited were the two honest staking-miners’ slashed deposits (community + Parity); the community operator has already been reimbursed via Ref 1915.

Era 2219’s reward accrued in full for its ~3.33-day duration.

One follow-up remains open: era 2219’s ~115,350 DOT self-stake incentive budget is not payable (the transition era has zero incentive weights as expected and the first payable era is 2220) and defaults to the DAP buffer in ~84 days unless redistributed. See “Era-2219 incentive budget” below.

Impact

  • Validator set frozen at the era-2219 set, including the 67 since-chilled validators, which remained active because no rotation occurred.
  • Era frozen (ActiveEra = 2219, CurrentEra = 2220), and with it:
    • staking rewards did not finalize or accrue to a new era;
    • unbonding / withdrawals did not progress (the unbonding period is counted in eras).
  • Repeated slashing of honest election miners’ deposits on each retry cycle:
    • Parity miner 16ciP5rjt4Yqivi1SWCGh7XsA8BDguV4tnTuyr937u2NME6h;
    • community miner 15qdfNWg9rgVYY9NH18R6F8hYGbKjQRVG5pqqPjzQEQF28n3
  • No consensus, parachain, or balance-transfer impact.

Timeline

When Event
2026-06-12 Ref 1890 enacted: “Staking: set min validator bond to 10K DOT” — MinValidatorBond 0 → 10,000 DOT. The proposal explicitly warned that “pre-existing under-bonded validators will face significant risk of chilling.”
2026-06-29 Ref 1909 enacted: “Staking Parameter & Budget Configuration Update” — ChillThreshold → 32%, MaxCommission → 0, DAP budget split (45.2% staker / 22.6% self-stake incentive / 32.2% buffer), concave self-stake incentive weight. MinimumScore left unchanged.
2026-06-29 With the chill_other gate now open, a (fully legitimate) chilling wave removes 67 under-bonded validators from the candidate set.
2026-06-29 A bot runs force_apply_min_commission across all validators, clamping every commission to 0%. Commission is captured at election time, so this applies from the next snapshot: era 2219 (already elected) still charged its original commission (~24% avg); 0% applies from era 2220.
2026-06-30 05:36 CET A new election starts and EPMB enters the Signed phase. Parity and community miners submit solutions; all are rejected ScoreTooLow and slashed. The unsigned solution is rejected for the same reason and EPMB falls back to the Signed phase. Election stuck at round 238 / era 2220.
2026-06-30 07:57 CET Parity staking-miner monitoring raises an alarm in the dedicated Matrix room: balance critically low.
2026-06-30 08:50 CET In the Polkadot Validator Lounge Matrix room, ech0.re reports the stuck election and his slashed miner. Thanks again ech0.re!
2026-06-30 09:39 CET Root cause identified and posted in the same room; mitigation drafted and dry-run-verified on a chopsticks fork of live Asset Hub.
2026-06-30 15:35 CET Remediation referenda submitted (Polkadot 1914 + Fellowship 569).
2026-06-30 17:00 CET Public communication on x.com and 110 channels (Telegram, Element, Slack, email).
2026-07-03 07:01 CET Ref 1914 enacted → MinimumScore.minimal_stake lowered to 500k DOT (sum_stake / sum_stake_squared unchanged).
2026-07-03 07:10 CET Election unstuck: MultiBlockElection.Round 238 → 239; a signed solution Registered and Rewarded (not slashed). Winning minimal_stake 675,132 DOT, which clears the 500k floor and is still below the old 789,555.
2026-07-03 09:36 CET Era-2220 validator set delivered to the relay (StakingAhClient.NextSessionChangesValidators = 2220, Mode = Active, Session.CurrentIndex = 13372).
2026-07-03 13:36 CET Validators activated on the relay; AH ActiveEra rotated 2219 → 2220 (CurrentEra = 2221, session 13373, ErasStakersOverview[2220] = 600). Era 2219 ran ~3.33 days and finalized with ErasValidatorReward[2219] = 230,699 DOT, exactly proportional to its duration at the new 45.2% staker share. Its 115,350 DOT incentive budget has zero weights and is not payable (see “Reward changes”).
2026-07-03 13:48 CET Resolution communicated on x.com and the same 110 channels.
2026-07-03 13:58 CET Set healing: round 239 queues minimal_stake 776,635 (up from 675,133), close to the old 789,555 floor already. Unsigned/OCW fallback verified viable (≈731k, +46% over the floor).
2026-07-04 13:36 CET Era 2220 closes normally (~24h); cadence fully restored, era 2221 active. First full era under the new split: staker reward ErasValidatorReward[2220] = 69,196 DOT (45.2%); first payable self-stake incentive ErasValidatorIncentiveBudget[2220] = 34,598 DOT (22.6%, ErasSumValidatorIncentiveWeight[2220] > 0); 0% commission.
2026-07-05 13:36 CET Era 2221 closes, steady state confirmed (second consecutive ~24h era): staker reward 69,235 DOT, self-stake incentive 34,617 DOT (payable). Set still healing: total elected stake 832.9M DOT (up from 811.4M in era 2220; era 2222 active at 838.0M), recovering toward the pre-chill ~862M.

Root cause

Every election solution must clear a fixed quality floor, MinimumScore, and that floor was the binding constraint here: the one check that actually failed. A permissionless chill_other wave had removed enough heavily-nominated validators that the best solution still achievable scored just below the floor on its dominant minimal_stake axis, so every submission was rejected and the era could not rotate.

In detail:

  1. Ref 1890 set MinValidatorBond = 10,000 DOT, defining who can be removed from the active validator set (self-bond < 10k).

  2. Ref 1909 set ChillThreshold = 32%, opening the permissionless chill_other density gate: ChillThreshold × MaxValidatorsCount < current validator count, i.e. 0.32 · 2000 = 640 < 1320. Before this, chill_other against validators was effectively closed. Note that chill_other keys purely on self-bond (ledger.active < MinValidatorBond).

  3. 67 of the 600 validators elected in era 2219 had self-bond below 10k DOT (the majority at zero) and were chill_other-ed off the candidate set on 2026-06-29.

  4. Despite tiny self-bond, those 67 carried ~96.6M DOT of nominator backing. With them gone, the election has to backfill 67 slots with compliant but less-backed validators, and the achievable minimal_stake (stake behind the least-backed elected validator) dropped sharply:

    minimal_stake vs floor
    MinimumScore floor 789,555 DOT —
    Era 2219 1,160,585 DOT +47%
    Best achievable (rejected) 666,297 DOT −15.6% → ScoreTooLow

  5. ElectionScore is compared lexicographically with minimal_stake dominant, so 666,297 < 789,555 means every solution is ScoreTooLow: the submitter is slashed and nothing is queued. The other two axes passed (sum_stake 811M > 565.6M floor; sum_stake_squared 1.13e35 < 1.87e35 floor); only minimal_stake failed.

  6. With nothing queued, the runtime reverts to the Signed phase and retries indefinitely (Continue fallback). The era never rotates, and the set, rewards and unbonding freeze with it.

The floor was set once, to 70% of the average of the last 30 pre-AHM relay-chain elections (snapshot at relay block 28,207,264). It is static by design: a deliberate, governance-controlled backstop (next section). It does not auto-adapt, so any change that moves the achievable score must be paired with an explicit update to it.

Why the minimum score exists (and why we keep it)

The MinimumScore floor is not the bug here. It did its job, and we want to keep it. Election miners are permissionless and the chain cannot trust their inputs, so the floor guarantees that no solution materially worse than a known-sane baseline is ever enacted, regardless of who submits (a signed miner, the unsigned/OCW fallback, or no one) and even if every solution in a round is bad.

That is exactly what happened in this incident: after the chilling wave the best achievable honest solution, signed and OCW alike, fell below the baseline, and the floor paused rotation rather than locking in a weakened set. Pausing is the intended outcome. Rewards and unbonding get delayed, not lost; enacting a materially worse validator set would be the genuinely bad result.

The floor also caps the one adversarial case the OCW fallback does not cover: a full solution that favours the submitter’s own validators can out-score the unsigned fallback (a partial, 4-page solution) while still being sub-optimal. The floor bounds how far that can go.

So neither the floor’s existence nor its static-ness is what went wrong. The floor worked; our process needs improvements.

Why it wasn’t detected

  • The referendum that moved the achievable score did not move the floor. Lowering ChillThreshold to 32% predictably enabled a chilling wave of under-bonded validators carrying large nominator backing, and removing them necessarily lowers the achievable minimal_stake. Ref 1909 should have bundled a matching SetMinUntrustedScore. Nothing flagged the mismatch.
  • The floor is static by design. A design choice we might revisit (see “What we are changing”), but not the cause: it only dictated how fast we could recover.
  • No simulation modeled the interaction. Three individually-reasonable factors (minimum self-stake, permissionless chilling, a fixed election floor) combined into a stall. Each looked safe alone; the combination was never tested against a realistic election.

Remediation

Immediate fix — lower the floor (Ref 1914)

Lower only the minimal_stake axis of MinimumScore, from 789,555 to 500,000 DOT, so the achievable solution (666,297 DOT) clears the floor and the election can finalize. sum_stake and sum_stake_squared already passed and were left unchanged.

The fix went through the Fellowship fast-track, which requires two referenda in parallel:

This was a stopgap to restore liveness. As expected, minimal_stake is already recovering naturally in the eras since enactment (675,132 → 776,635), as the active set converges on validators that meet the 10k self-stake minimum and carry proper nominator backing.

Worth noting: the new floor is compatible with the unsigned (OCW) fallback, not just with signed solutions. The unsigned miner submits a partial solution (MinerPages = 4 of 32 voter pages), which scores lower than a full signed one — but only marginally, because the top 4 voter pages hold ~95.8% of the electing stake. On a live round-239 snapshot, the unsigned minimal_stake came out at ≈ 731,100 DOT vs ≈ 776,783 for the full solution, clearing the 500k floor by ~46%. If signed miners drop out, the network’s own validators can therefore still complete the election via the unsigned phase. During the incident this safety net was also gone: the old 789,555 floor sat above even the full signed solution (666–675k), so the lower-scored unsigned solution failed too, leaving no fallback at all.

Reimbursement of the slashed community staking-miner

The slashed miners (ech0.re’s 15qdfNWg9rgVYY9NH18R6F8hYGbKjQRVG5pqqPjzQEQF28n3 and Parity’s 16ciP5rjt4Yqivi1SWCGh7XsA8BDguV4tnTuyr937u2NME6h) submitted honest solutions that were rejected only because the floor had become unreachable, through no fault of their own. ech0.re has already been refunded via a dedicated reimbursement referendum, Polkadot Referendum 1915, covering the deposits slashed during the stall.

Reward changes: what actually happened

Two distinct things affected rewards. Only the first is caused by this incident, and neither loses emission.

1. No emission was lost to the extended era. Era 2219 ran 3 days 8 hours (~3.33× a normal era) and accrued proportionally for that duration. The figures reconcile exactly:

  • Total era-2219 emission ≈ 510,396 DOT, split per the DAP allocation: staker 230,699 (45.2%, on-chain ErasValidatorReward), validator self-stake incentive 115,350 (22.6%, on-chain ErasValidatorIncentiveBudget, exactly 2:1 to the staker share), buffer 164,348 (32.2%).
  • Daily emission is essentially unchanged at ~153,132 DOT/day in both regimes: 130,162 / 0.85 before the split (eras 2214–2217), and era 2219’s 510,396 / ~3.333 days lands on the same figure.

2. Per-day staker rewards are lower now — a deliberate governance change, not the stall. Ref 1909’s dap.setBudgetAllocation re-split the emission: the direct staker share dropped from ~85% to 45.2%, a new 22.6% validator self-stake incentive was introduced, and the DAP buffer rose to 32.2%. A normal-era staker payout therefore fell from ~130,162 to ~69,196 DOT/day (confirmed on era 2220, the first full era under the split — see timeline). This is by design, approved by the community via Ref 1909, and mechanically unrelated to the election stall: both changes shipped in Ref 1909’s batch, but the reward split has nothing to do with the ChillThreshold change that triggered the chill_other wave.

Three more transition details across era 2219 → 2220:

3. Commission. Validators still charged commission for era 2219 (avg ~24%, min 10%, captured at election time); the 0% clamp applies from era 2220.

4. Self-stake incentive. Works as designed. It takes effect one era after the referendum, so the first era with incentive weights, and hence a payable incentive, is era 2220. Era 2219, the transition era, has zero weights, so its budget is not payable. The only unfortunate knock-on of the ~3.33-day era: (a) the first incentive payout (era 2220’s, paid when that era closes) arrived ~2 days late; and (b) the ~115,350 DOT that accrued to the incentive pot during the long transition era is not payable and returns to the DAP buffer at HistoryDepth pruning (start of era ~2304, ~84 days out, via UnclaimedRewardHandler = Dap), unless we take action (next section).

5. Unbonding. Withdrawal maturity is counted in eras. A chunk is released only once active_era reaches its unlock era, and active_era was frozen at 2219. Nothing was lost, only delayed: once eras rotated again (post Ref 1914), queued unbondings resumed maturing.

Era-2219 incentive budget: how to handle the ~115,350 DOT

Whether to redistribute era 2219’s unpayable incentive budget to stakers, instead of letting it default to the DAP buffer, is an open question, with the ~84-era pruning window as the deadline.

I believe the best outcome is to redistribute the amount to validators in a fair way. A good suggestion, posted by @CoinStudio in the Polkadot Validator Lounge on Matrix, is to redistribute the ~115k DOT equally among the 600 validators active in era 2219.

Any other suggestion on how to best handle this is very much welcome.

What we are changing

This incident was avoidable, and we apologize for it.

The fix enacted by Ref 1914 unblocked the chain, but the underlying gaps — process, testing, and recovery when the minimum score cannot be met — remain, and we intend to close them.

1. Testing, simulation, and a pre-release checklist

Parameter-change referenda that touch the election (bonds, chilling thresholds, validator count, score floors, incentives) must be fully simulated against a realistic, current snapshot before enactment, including second-order effects.

As a result, Ref 1909 should have bundled the corresponding MinimumScore reduction as an outcome of that simulation.

We will formalize this into a pre-release checklist that gates every sensitive staking change, automated as far as possible. The scriptable items should run automatically: the forked-chain dry-run of the resulting election, an assertion that the achievable score still clears the configured MinimumScore with margin, a check that any change moving the achievable score ships its paired SetMinUntrustedScore. Today the process is too manual, and therefore error-prone. The goal: a change like Ref 1909 shipped without its floor adjustment fails the checklist.

Tracking issue: polkadot-sdk#12564.

2. Better monitoring and alerting

Detection today leans too heavily on the miner’s own signals. The Parity staking-miner raises automatic alarms (solution-not-submitted, balance-too-low, and related failure modes) surfaced on Matrix, with a dedicated Grafana dashboard for the Parity miner’s activity. Era and election health are otherwise watched manually. This incident was accordingly flagged by symptoms — miners slashed, balances draining — rather than by a direct signal on election or era health.

We will add richer real-time, chain-level alerts that fire on the failure conditions directly, independent of any miner:

  • election not progressing: Round not advancing, phase looping back to Signed, repeated ScoreTooLow rejections;
  • era not rotating within its expected window;
  • achievable election score approaching the MinimumScore floor (early warning, before it breaches);
  • abrupt drops in candidate/validator count, or chill_other waves;
  • no queued solution as a phase deadline nears.

Tracking issue: polkadot-sdk#12568.

3. Minimum score: process, and safer deadlock recovery

We keep the minimum score as a deliberate, governance-controlled backstop, with a better process around it and, eventually, a better way to recover.

Process. Any referendum that changes parameters affecting the achievable election score must include the matching SetMinUntrustedScore in the same referendum, with the value derived from the simulated election of §1. We will provide tooling to compute the appropriate floor (e.g. a fixed margin below the simulated achievable score for the post-change set) so proposers can bundle the adjustment confidently.

Changes in the logic. We should make recovery from a score-too-low deadlock faster without lowering safety. An automatic decay of the floor was considered and rejected: too slow and it is no better than a root call; too fast and an adversary can deliberately stall the election to drive the floor down, then land a sub-optimal, self-favouring solution. Ideally a stall would be detected automatically and resolved within hours, without manual intervention. Can we achieve that safely? Or is this precisely a case where governance should stay in the loop? I don’t have an answer for that now. Directions we will explore:

  • recovery that refreshes inputs without weakening the backstop, e.g. re-snapshotting on a detected stall;
  • deriving MinimumScore from a slow-moving trailing statistic of recent accepted scores — deliberately not submitted ones, which are permissionless and could be spammed low to poison the floor. Note this mainly addresses the floor going stale, not this incident’s trigger;
  • even without automatic recovery, emitting a clear on-chain event once the election has failed to finalize for N rounds, so monitoring can trigger a governance fast-track quickly.

Tracking issue: polkadot-sdk#12545.

4. Miner client guard

The polkadot-staking-miner has been updated (PR #1291) to skip submitting a solution whose score is below the on-chain MinimumScore. This stops honest miners from being slashed for submitting sub-floor solutions — the deposit-draining seen in this incident — independent of the runtime-side changes above.

5. Invulnerable miners

SDK PR #10454 already changed the runtime so that an invulnerable submitter is no longer auto-expelled on a bad solution. We may additionally add one or more miners to the invulnerable list.

2

Validators’ losses always seem to receive attention and compensation, while the losses suffered by nominators in past incidents have received no action at all.