Date: 30 September 2025
Duration: 15:06:12 – 18:41:18 UTC
Impact: Polkadot → Ethereum transfers unavailable
Summary
On 30 September 2025, the Polkadot runtime upgrade 1.7.1 was enacted. The runtime upgrade Wasm was uploaded to Polkadot Asset Hub, but not to Polkadot Bridge Hub. This mismatch caused Snowbridge to reject messages due to unexpected runtime code, resulting in an outage for transfers from Polkadot to Ethereum. 6 transactions from Hydration to Ethereum were impacted. No permanent fund loss occurred.
Root Cause
Code Change
A change was introduced on Bridge Hub and Asset Hub to allow Asset Hub to send unpaid execution messages to Bridge Hub:
https://github.com/paritytech/polkadot-sdk/pull/8599
This change was done to simplify fee handling in Snowbridge V2. It changes Snowbridge messages from Asset Hub to unpaid execution messages. A corresponding change on Bridge Hub allows these unpaid messages, which were previous paid.
When Asset Hub was upgraded to 1.7.1, the Snowbridge messages changed to unpaid messages, while Bridge Hub (not upgraded) expected paid messages. Bridge Hub consequently rejected the message.
Runtime Upgrade
The cause of the mismatch in system parachain upgrades lies in how runtime upgrades are executed.
When a referendum executes, it does not directly perform a chain upgrade. Instead, the referendum grants permission for the upgrade to occur. The actual chain upgrade requires a manual upload of the runtime Wasm (binary runtime code) ****to the chain.
-
This upload is permissionless — any network participant can submit it.
-
Once uploaded, the chain schedules the upgrade and executes it automatically within about one hour.
In this case:
-
The referendum for runtime 1.7.1 executed on both Asset Hub and Bridge Hub.
-
The Wasm was uploaded on Asset Hub, triggering its upgrade at 15:06:12 UTC. Asset Hub sends unpaid Snowbridge transactions to Bridge Hub.
-
The Wasm was not uploaded on Bridge Hub at the same time, leaving it on the previous runtime. Bridge Hub still expects paid Snowbridge transactions from Asset Hub.
-
This mismatch between Asset Hub and Bridge Hub runtimes caused Snowbridge’s runtime verification logic to fail, blocking all outbound transfers.
-
The Snowbridge team uploaded the missing Wasm to Bridge Hub at 17:40:30 UTC, allowing the upgrade to execute by 18:41:18 UTC, restoring services.
Timeline
-
15:06:12 UTC – Polkadot Asset Hub upgraded to runtime 1.7.1.
-
15:06–18:41 UTC – Transfers from Polkadot → Ethereum failed.
-
17:40:30 UTC – Snowbridge team uploaded the missing Bridge Hub runtime Wasm.
-
18:41:18 UTC – Polkadot Bridge Hub runtime upgrade executed. Snowbridge services restored.
Impact
-
6 failed transfers during the outage window. All failures originated from Hydration → Ethereum transfers:
-
https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-8435ced06684d769562d3bcdc538c6174bc8803c
-
https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-f79593c73f30bf4c4e4f91debc58e2ce60cd5173
-
https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-7c2276b241ea8a0eacc0ef52eb9e08ba876a334d
-
https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-148e4b5922613bc3005c008873ac50b37fc3aee4
-
https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-de8eae6e1024e22e76c1405389e6ae38dc049261
-
https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-16d3f6aaf7b96f7f6e7ec8dd6222d15a67b78e26
-
Actions Taken
-
Immediately set Snowbridge UI to maintenance mode.
-
Investigated the error and found that Bridge Hub has not been upgraded.
-
Uploaded the missing Bridge Hub runtime Wasm.
-
Informed partner UIs (Hydration, Turtle) of the outage.
-
Monitored and tracked all failing transactions.
-
Notified the wider community of the incident.
Resolution
Once the Bridge Hub upgrade executed (approx. 1 hour delay due to governance process), Snowbridge resumed normal operations. Failed transaction will be replayed in ref https://polkadot.subsquare.io/referenda/1765.
Corrective Actions
-
Investigation Into How This Occurred: We need to understand how an upgrade was only partially applied, and how to prevent this from happening again, both on a human and process level, as well as technical changes (i.e. specificing version dependencies between runtimes, .etc)
-
Referendum to Replay Transactions: A governance referendum has been proposed to replay the 6 failed Hydration transfers, ensuring that affected users are fully restored.
-
Expanded Alarms: Snowbridge has an extensive suite of alarms to monitor our service. Our alarms did not catch this specific incident, and will be improved to do so. Monitoring and alerting will be extended to specifically detect mismatches between Asset Hub and Bridge Hub Wasm versions in the future.
-
Upgrade Responsibility: The Snowbridge team will take responsibility for coordinating and ensuring that both Asset Hub and Bridge Hub Wasms are uploaded for all future runtime upgrades, preventing recurrence.
-
Upgrade Process: When the upgrade is complex or significant, we will more formally control the upgrade process by putting our app under maintenance mode, notifying our partners and communicating to the ecosystem of the upcoming upgrade.
-
More Resilient On-Chain Code: For future upgrades, ensure AssetHub and BridgeHub can be upgraded independently, with automatic transaction replays on failure.
