PSA: Disruption To Kusama Validators' Self-Stake and Rewards

Hi everyone. Writing this post to give an update about a bug that was recently discovered in the Kusama Network after AHM was executed.

Throughout the migration, the old pallet-stakin in the Kusama RC was retired and replaced with pallet-staking-async living in Kusama AH. The same flow is scheduled repeat in Polkadot Nov 4th.

Thanks to a number of reports for Kusama validators we identified a bug in pallet-staking-async that causes the following issue: In most cases, self-stake of validators was not being correctly recorded onchain. The main consequence of this is that for a number of eras Kusama validators have not been rewarded (nor are slashable) for their self-stake in recent eras.

We identified this issue last Friday, and by Sunday, a fix was ready: [AHM/Staking] Allow own stake to be collected in all pages by kianenigma · Pull Request #10051 · paritytech/polkadot-sdk · GitHub

We are now proceeding with merging this fix in the Kusama runtime code, and propose a 1.9.3 runtime upgrade to fix the issue.

This problem will not exist in the upcoming Polkadot release on Nov 4th, the Polkadot AHM.

I want to thank the validators who reported the issue, and encourage them to, after the fix is enacted, calculate their “missing rewards” in the period since Kusama AHM was executed up until the fix, and seek a tip from the treasury.

Example

Without going into too much details, let’s consider one of the validators in Kusama: DNBrjMnPyjEeBsDUkyeWJ9ba6eVdaFHixUPM2MGgms4xcyK. This validator has a self-stake of ~159KSM (query=staking.ledger), which in most cases, should become active in the staking election. Looking at the final ExposureMetadata (query=staking.erasStakersOverview) of this validator in the active era, we can see that own is set to zero, which means to self-stake was recorded for this validator.

Related Work

10 Likes

Should every single validator do that? Do they all know how? Shouldn’t it be pushed to them automatically?

Once we have the fix in place, we can do a back of an envelope calculation and tip all of them. I am for now encouraging others to look into it, hopefully one community member will pick this up and calculate it. If not, I will do it myself :handshake:

2 Likes

The fix referendum has been executed: https://kusama.subsquare.io/referenda/608

3 Likes

Following referendas execution, Kusama Asset Hub has been updated to 1.9.3 on block:

I’ve put together a draft example of the self-stake missed rewards here Draft self-stake missing rewards calculation by paulormart · Pull Request #142 · turboflakes/subxt · GitHub If someone else would like to verify would be great.

From my calculations the missed self-stake rewards resulted on 401.651409929767 KSM for 876 validators.

1 Like

Thanks for the calculation @paulormart! If the validator community is inclined to do so, will support a proposal to refund them. Although, the values might be small enough that validators might be okay to write it off as a one-off incident, as over the course of an annual payout, it will be a tiny fraction.

1 Like