# Snowbridge Incident Report – Polkadot Runtime Upgrade 1.7.1 **URL:** \ **Category:** Miscellaneous\ **Tags:** bridge\ **Created:** [October 1, 2025, 9:54am UTC](https://forum.polkadot.network/t/snowbridge-incident-report-polkadot-runtime-upgrade-1-7-1/15144 "2025-10-01T09:54:53Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![claravanstaden](https://dub1.discourse-cdn.com/flex005/user_avatar/forum.polkadot.network/claravanstaden/32/6575_2.png) [@claravanstaden](https://forum.polkadot.network/u/claravanstaden)\ **Post date:** [October 1, 2025, 9:54am UTC](https://forum.polkadot.network/t/snowbridge-incident-report-polkadot-runtime-upgrade-1-7-1/15144/1 "2025-10-01T09:54:53Z")
**Date:** 30 September 2025 **Duration:** 15:06:12 – 18:41:18 UTC **Impact:** Polkadot → Ethereum transfers unavailable ## Summary On 30 September 2025, the Polkadot runtime [upgrade 1.7.1 was enacted](https://polkadot.polkassembly.io/referenda/1756). The runtime upgrade Wasm was uploaded to Polkadot Asset Hub, but not to Polkadot Bridge Hub. This mismatch caused Snowbridge to reject messages due to unexpected runtime code, resulting in an outage for transfers from Polkadot to Ethereum. 6 transactions from Hydration to Ethereum were impacted. No permanent fund loss occurred. ## Root Cause ### Code Change A change was introduced on Bridge Hub and Asset Hub to allow Asset Hub to send unpaid execution messages to Bridge Hub: [https://github.com/paritytech/polkadot-sdk/pull/8599](https://github.com/paritytech/polkadot-sdk/pull/8599) This change was done to simplify fee handling in Snowbridge V2. It changes Snowbridge messages from Asset Hub to unpaid execution messages. A corresponding change on Bridge Hub allows these unpaid messages, which were previous paid. When Asset Hub was upgraded to 1.7.1, the Snowbridge messages changed to unpaid messages, while Bridge Hub (not upgraded) expected paid messages. Bridge Hub consequently rejected the message. ### Runtime Upgrade The cause of the mismatch in system parachain upgrades lies in how runtime upgrades are executed. When a referendum executes, it does not directly perform a chain upgrade. Instead, the referendum grants permission for the upgrade to occur. The actual chain upgrade requires a manual upload of the runtime Wasm (binary runtime code) \*\*\*\*to the chain. - This upload is permissionless — any network participant can submit it. - Once uploaded, the chain schedules the upgrade and executes it automatically within about **one hour.** In this case: - The referendum for runtime 1.7.1 executed on both Asset Hub and Bridge Hub. - The Wasm was uploaded on **Asset Hub** , triggering its upgrade at 15:06:12 UTC. Asset Hub sends unpaid Snowbridge transactions to Bridge Hub. - The Wasm was **not uploaded** on Bridge Hub at the same time, leaving it on the previous runtime. Bridge Hub still expects paid Snowbridge transactions from Asset Hub. - This mismatch between Asset Hub and Bridge Hub runtimes caused Snowbridge’s runtime verification logic to fail, blocking all outbound transfers. - The Snowbridge team uploaded the missing Wasm to Bridge Hub at 17:40:30 UTC, allowing the upgrade to execute by 18:41:18 UTC, restoring services. ## Timeline - **15:06:12 UTC** – [Polkadot Asset Hub upgraded to runtime 1.7.1.](http://assethub-polkadot.subscan.io/extrinsic/9784456-0) - **15:06–18:41 UTC** – Transfers from Polkadot → Ethereum failed. - **17:40:30 UTC** – [Snowbridge team uploaded the missing Bridge Hub runtime Wasm](https://bridgehub-polkadot.subscan.io/extrinsic/6102738-2). - **18:41:18 UTC** – [Polkadot Bridge Hub runtime upgrade executed.](https://bridgehub-polkadot.subscan.io/extrinsic/6102997-0) Snowbridge services restored. ## Impact - **6 failed transfers** during the outage window. All failures originated from **Hydration → Ethereum** transfers: ## Actions Taken - Immediately set Snowbridge UI to maintenance mode. - Investigated the error and found that Bridge Hub has not been upgraded. - Uploaded the missing Bridge Hub runtime Wasm. - Informed partner UIs (Hydration, Turtle) of the outage. - Monitored and tracked all failing transactions. - [Notified](https://x.com/_snowbridge/status/1973112209833337053) the wider community of the incident. ## Resolution Once the Bridge Hub upgrade executed (approx. 1 hour delay due to governance process), Snowbridge resumed normal operations. Failed transaction will be replayed in ref [https://polkadot.subsquare.io/referenda/1765](https://polkadot.subsquare.io/referenda/1765). ## Corrective Actions - **Investigation Into How This Occurred** : We need to understand how an upgrade was only partially applied, and how to prevent this from happening again, both on a human and process level, as well as technical changes (i.e. specificing version dependencies between runtimes, .etc) - **Referendum to Replay Transactions** : A governance referendum has been proposed to replay the 6 failed Hydration transfers, ensuring that affected users are fully restored. - **Expanded Alarms** : Snowbridge has an extensive suite of alarms to monitor our service. Our alarms did not catch this specific incident, and will be improved to do so. Monitoring and alerting will be extended to specifically detect mismatches between Asset Hub and Bridge Hub Wasm versions in the future. - **Upgrade Responsibility** : The Snowbridge team will take responsibility for coordinating and ensuring that both Asset Hub and Bridge Hub Wasms are uploaded for all future runtime upgrades, preventing recurrence. - **Upgrade Process:** When the upgrade is complex or significant, we will more formally control the upgrade process by putting our app under maintenance mode, notifying our partners and communicating to the ecosystem of the upcoming upgrade. - **More Resilient On-Chain Code:** For future upgrades, ensure AssetHub and BridgeHub can be upgraded independently, with automatic transaction replays on failure. ---
**Author:** ![acatangiu](https://dub1.discourse-cdn.com/flex005/user_avatar/forum.polkadot.network/acatangiu/32/540_2.png) [@acatangiu](https://forum.polkadot.network/u/acatangiu)\ **Post date:** [October 1, 2025, 10:30am UTC](https://forum.polkadot.network/t/snowbridge-incident-report-polkadot-runtime-upgrade-1-7-1/15144/2 "2025-10-01T10:30:25Z")
Thank you for the report Clara! > Once the Bridge Hub upgrade executed (approx. 1 hour delay due to governance process), Snowbridge resumed normal operations. Failed transaction will be replayed in ref [https://polkadot.subsquare.io/referenda/1765](https://polkadot.subsquare.io/referenda/1765). Can you provide some tooling or guide for people to check that the referenda replays the failed transactions and nothing more? (I knew what I was doing and it still was hard to check, especially diffing the accounts and amounts) ---
**Author:** ![claravanstaden](https://dub1.discourse-cdn.com/flex005/user_avatar/forum.polkadot.network/claravanstaden/32/6575_2.png) [@claravanstaden](https://forum.polkadot.network/u/claravanstaden)\ **Post date:** [October 1, 2025, 11:47am UTC](https://forum.polkadot.network/t/snowbridge-incident-report-polkadot-runtime-upgrade-1-7-1/15144/3 "2025-10-01T11:47:43Z")
> [@claravanstaden](#): > > [https://polkadot.subsquare.io/referenda/1765](https://polkadot.subsquare.io/referenda/1765) ## Verifying the referendum To verify the referendum, follow these steps. View the referendum at: [https://polkadot.subsquare.io/referenda/1765?tab=call](https://polkadot.subsquare.io/referenda/1765?tab=call) Click on the `View Detail` icon next to the call details: ![Screenshot 2025-10-01 at 13.27.57](https://europe1.discourse-cdn.com/flex005/uploads/polkadot2/original/2X/a/a7c7d11502968d600927968d70950bc509835d41.png) Click on the `+` icon to view the entire call. ![Screenshot 2025-10-01 at 13.29.22](https://europe1.discourse-cdn.com/flex005/uploads/polkadot2/original/2X/2/231561eaeeef6336cedbe53b125152164ead607a.png) You will see the call sends a `Transact` message to Asset Hub. Scroll down to the `Transact` bytes, and click on `Download` to download the full transact payload. ![Screenshot 2025-10-01 at 13.29.50](https://europe1.discourse-cdn.com/flex005/uploads/polkadot2/original/2X/9/90c48f1427211b62b1777ec9d685cc1cfd9c6216.png) Copy the downloaded hex and visit Polkadot Asset Hub, to decode the message bytes: > **[Polkadot/Substrate Portal](https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Frpc-asset-hub-polkadot.luckyfriday.io#/extrinsics/decode)** ![Screenshot 2025-10-01 at 13.30.56](https://europe1.discourse-cdn.com/flex005/uploads/polkadot2/original/2X/5/5797bda20d14b12bba1a78dde1dc655f1229ad97.png) Verify each `ExportMessage` against the list of failed XCM messages listed above in the incident report. They should match, exactly. Affected transactions and amounts: | Token | Amount | Beneficiary | Topic | Transaction | | --- | --- | --- | --- | --- | | sUSDe | 16,716 | 0x9117900a3794ad6d167dd97853f82a1aa07f9bbc | 0xe4cff6bf2217eb4cf9332d2daee1ada70b405402414a2249a6e9b42ab759f93f | [Link](https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-8435ced06684d769562d3bcdc538c6174bc8803c) | | tBTC | 0.2508307657288558 | 0x601d579ecd0464a1a090ceef81a703465a1679cd | 0xd289d29c0ccbca0fe47be2a0bf8d09af3a90d719ce62129d75714a342750b6e4 | [Link](https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-f79593c73f30bf4c4e4f91debc58e2ce60cd5173) | | SKY | 90413.71054397589 | 0x601d579ecd0464a1a090ceef81a703465a1679cd | 0xf701fb349a04e4c923e26aab4e0288975d904507cdc32a3d3bdab8105507c736 | [Link](https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-7c2276b241ea8a0eacc0ef52eb9e08ba876a334d) | | AAVE | 33.044703802651993696 | 0x2265a7503597ab32bab72eaa186e6329fb7b68f3 | 0xc8864869cd4ed5921d5cc251290357ffb24f905e1a475a2ba6c9ecd96c55df71 | [Link](https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-148e4b5922613bc3005c008873ac50b37fc3aee4) | | AAVE | 212.116067921877821839 | 0xa9c415d6881e1a992861a7fa6bef3ed4736152c2 | 0xe259e4fb1c24f7cf4e6d2a9d50e13794de5fd6863083addc3d55ddff3b3d58cd | [Link](https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-de8eae6e1024e22e76c1405389e6ae38dc049261) | | ETH | 0.35 | 0xad8d4c544a6ce24b89841354b2738e026a12bca4 | 0x1ae83a0cba8f448c466fb0863fc25827b6978b7c3b3f93785184412cb2632e31 | [Link](https://bridgehub-polkadot.subscan.io/xcm_message/polkadot-16d3f6aaf7b96f7f6e7ec8dd6222d15a67b78e26) |