Async Backing - Development Updates

[joyce]

Dear Community,

We are thrilled to announced that we are hitting constant 6s blocks in test networks. An active example using Async Backing is the System Parachain Asset Hub on Rococo

After fighting around with a critical bug that was blocking our launch, we’re back in the fast lane, speeding towards the launch of Asynchronous Backing on Kusama.

If you want to be on the loop with our development, you can either have a look at our enablement checklist or just simply hit the notification button of this thread, we will keep a regular update here.

In the meanwhile, if you haven’t so, you can make yourself familiar with Asynchronous Backing:

• Last year Sophia Gold gave a presentation on sub0 in Lisbon introducing Asynchronous Backing
• On the Polkadot Wiki you can find all the terminologies and concepts
• Here you will find a guide in order to get your Parachain compatible for Asynchronous Backing

[joyce]

If you’re facing problems while implementing/testing async. backing, please open an issue on Github and tag it with T16-async_backing. We are happy to learn about your issues and gain feedback in order to improve it.

[alice_und_bob]

For which runtime release is Async Backing for Kusama currently queued?

[joyce]

We currently can’t tell that since we’re facing some bugs that need to be fixed before moving on, but we will keep giving updates on this post and will inform you soon enough about the release on Kusama.

[joyce]

… and we keep on rolling
Networking bugs fixed, that were causing some fractions on the block producing time
Kusama runtime fix merged and included for the next fellowship release

Next up

Activate Async Backing on Kusama

After the upcoming Fellowship runtime with a referendum we will activate async backing on Kusama, it will be safe for governance to enable 6s block times for parachains (if that’s what they want).

Enable Async Backing on Polkadot

We were able to identify the one remaining issue with async backing in legacy mode on Kusama. With this, we are ready to enable async backing on Polkadot as well. Meaning, after the upcoming Fellowship runtime release is enacted, 6s block times are only a governance motion away. That motion should be proposed after we saw it working flawlessly on Kusama.

VALIDATORS PLEASE UPGRADE

In order to make the launch of async backing successful on Polkadot, we need your coopearation: Async backing is a new feature which is only supported by recent node versions. We recommend to run the latest version (1.8 at the time of writing). Running older versions, poses the risk of receiving less era points.

Please, feel free to reach out if you have any questions! The next update will be posted in 4 weeks again on this thread

[joyce]

The Kusama runtime upgrade is now out for voting including the activation of Async Backing. If this referenda gets accepted by the community and enacted, we have a massive performance boost on Kusama (and also on Polkadot). With 4x more execution time (from 500ms to 2s) and 2x reduced parablock time (from 12s to 6s)

In order to use Async Backing, make sure your chain has the pre-requisites set up and follow the guide provided here for the activation.

If you have any questions or are facing issues, reach out to us by creating an issue on Github and labeling it with T16-async_backing.

[joyce]

CTA: All collators should upgrade to the polkadot-sdk v1.8.0 (or higher)

As soon as asynchronous backing is activated on Kusama, you might receive an error of the parachain runtime panicking. The error is harmless and can be ignored, the Parachains will continue producing blocks at a 12s rate.

The logs for example will look like this:

2024-04-06 10:26:48.019 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] [process_inherent_data] bitfields.len(): 2, backed_candidates.len(): 1, disputes.len() 0    
2024-04-06 10:26:48.019 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Size before filter: 738, candidates + bitfields: 737, disputes: 1    
2024-04-06 10:26:48.019 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Time weight before filter: 8569167896, candidates + bitfields: 8569167896, disputes: 0    
2024-04-06 10:26:48.019 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Max block weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-04-06 10:26:48.019 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block time weight: Weight(ref_time: 2000000000000, proof_size: 18446744073709551615)    
2024-04-06 10:26:48.019 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block size: 5242880    
2024-04-06 10:26:48.019 DEBUG tokio-runtime-worker runtime::inclusion-inherent: [Relaychain] Used max block weight: Weight(ref_time: 2000000000000, proof_size: 5242880)    
2024-04-06 10:26:48.020 DEBUG tokio-runtime-worker runtime::parachains::scheduler: [Relaychain] [occupied] now_occupied {<wasm:stripped>: <wasm:stripped>}    
2024-04-06 10:26:48.021 DEBUG tokio-runtime-worker runtime::system: [Relaychain] [14] 0 extrinsics, length: 1079 (normal 0%, op: 0%, mandatory 0%) / normal weight:Weight(ref_time: 0, proof_size: 0) (0%) op weight Weight(ref_time: 0, proof_size: 0) (0%) / mandatory weight Weight(ref_time: 11472299459, proof_size: 53188) (0%)    
2024-04-06 10:26:48.022  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #14 (0xf8ff…b379)    
2024-04-06 10:26:48.025 DEBUG tokio-runtime-worker aura::cumulus: [Parachain] Adjusted relay-chain slot to parachain slot relay_slot=Slot(285398668) para_slot=Slot(142699334) timestamp=Timestamp(1712392008000) slot_duration=SlotDuration(12000) relay_chain_slot_duration=6s
2024-04-06 10:26:48.025  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🙌 Starting consensus session on top of parent 0x3b69b2c9e2ee92eedc182652f8e7333a16d413320a1a9852b5d4a6517fabd3d8    
2024-04-06 10:26:48.026 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/s0me0ne/wrk/parity/tmp/polkadot-sdk/cumulus/pallets/parachain-system/src/lib.rs:1339:9:
no space left for the block in the unincluded segment    
2024-04-06 10:26:48.027  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-04-06 10:26:48.027  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-04-06 10:26:48.027  WARN tokio-runtime-worker basic-authorship: [Parachain] ❗ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x4af5e3 - <unknown>!rust_begin_unwind
    1: 0x3908a9 - <unknown>!core::panicking::panic_fmt::he960afd153cd3fd9
    2: 0x285fa5 - <unknown>!cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::maybe_drop_included_ancestors::h3718993d55b394d4
    3: 0x2b180a - <unknown>!frame_support::storage::transactional::with_transaction::h6fea0184c87af272
    4: 0xd851a - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::hd9ca3ce538c161f7
    5: 0xdde9d - <unknown>!environmental::local_key::LocalKey<T>::with::h74efd0e12fe59848
    6: 0x48125 - <unknown>!<bridge_hub_rococo_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h1a1f14c58a84e4c6
    7: 0x47f18 - <unknown>!<bridge_hub_rococo_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h1709560e0b80ff8f
    8: 0x2a0bd4 - <unknown>!<sp_runtime::generic::checked_extrinsic::CheckedExtrinsic<AccountId,Call,Extra> as sp_runtime::traits::Applyable>::apply::h50064e7addfca962
    9: 0x315de6 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::h97a9b8468f3be133
   10: 0x105f2e - <unknown>!BlockBuilder_apply_extrinsic. Dropping.    
2024-04-06 10:26:48.027 DEBUG tokio-runtime-worker runtime::xcmp-queue-migration: [Parachain] Lazy migration finished: item gone    
2024-04-06 10:26:48.027 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/s0me0ne/wrk/parity/tmp/polkadot-sdk/cumulus/pallets/parachain-system/src/lib.rs:265:18:
set_validation_data inherent needs to be present in every block!    
2024-04-06 10:26:48.027 ERROR tokio-runtime-worker aura::cumulus: [Parachain] err=Error { inner: Proposing

Caused by:
    0: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x4af5e3 - <unknown>!rust_begin_unwind
           1: 0x3908a9 - <unknown>!core::panicking::panic_fmt::he960afd153cd3fd9
           2: 0x390957 - <unknown>!core::option::expect_failed::heb11ec0037c97265
           3: 0x2805d4 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::h16b45a38e0172da5
           4: 0x1dfbb9 - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::hd014aa5a09a5f217
           5: 0x3162ad - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook::hd4efe0ffab0f3a1f
           6: 0x3163ad - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block::h2ad419b1d2d0bc1f
           7: 0x105fda - <unknown>!BlockBuilder_finalize_block
    1: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x4af5e3 - <unknown>!rust_begin_unwind
           1: 0x3908a9 - <unknown>!core::panicking::panic_fmt::he960afd153cd3fd9
           2: 0x390957 - <unknown>!core::option::expect_failed::heb11ec0037c97265
           3: 0x2805d4 - <unknown>!<cumulus_pallet_parachain_system::pallet::Pallet<T> as frame_support::traits::hooks::OnFinalize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_finalize::h16b45a38e0172da5
           4: 0x1dfbb9 - <unknown>!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::hd014aa5a09a5f217
           5: 0x3162ad - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::idle_and_finalize_hook::hd4efe0ffab0f3a1f
           6: 0x3163ad - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::finalize_block::h2ad419b1d2d0bc1f
           7: 0x105fda - <unknown>!BlockBuilder_finalize_block }
2024-04-06 10:26:48.679  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (3 peers), best: #14 (0xf8ff…b379), finalized #11 (0x3ec5…d206), ⬇ 1.4kiB/s ⬆ 0.9kiB/s    
2024-04-06 10:26:48.683  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (1 peers), best: #2 (0x3b69…d3d8), finalized #0 (0x2360…718f), ⬇ 0.2kiB/s ⬆ 0.1kiB/s

This is cause due to how the slot assignment is calculated, since with a slot duration of 12s and blocks building every 6s, you will get the same slot assigned on every second block and we don’t allow multiple blocks per slot, the runtime will panic. In order to avoid that, we have implemented a fix that ensures we don’t build multiple blocks per slot, which is included in the polkadot-sdk v1.8.0.

Please, upgrade your Collator nodes if you don’t want these error messages on your logs. In case you’re facing problems while upgrading so please open an issue on Github and label it with T16-async_backing.

[joyce]

It seems like we still have about 8% of validators behind on Polkadot.

Listed validators, please upgrade to at least >= 1.7.0, the Polkadot runtime upgrade 1.2 enables several new protocols which are a pre-requisite for async backing. If you don’t upgrade, you won’t be able able to participate in the backing stage and will get 0 rewards for it.

[joyce]

The new Parachains validation timeouts parameters were enacted on Kusama and will be enacted on Polkadot in some hours too.

Please mind that this change forces the PVFs to be recompiled on all validators at once and will cause a lag on candidate validation and block finalization.

This might also result in a temporary liveness issue for Parachains:

This behavior shouldn’t be worrisome and will restabilize itself after 5-15 minutes.

We apologize for the short notice and the inconvenience, but it is a required change to ensure maximum performance for Parachains wishing to fully utilize the abilities of asynchronous backing.

In order to avoid liveness issues in production networks, we are already working on a solution to ensure that only changes to preparation-related parameters triggers re-compilation. A ticket for this can be found here.

[joyce]

With the execution of Referenda #688 yesterday, Async Backing is finally live on Polkadot If any bugs or questions arise, feel free to open a ticket on Github or raise a question on Stackexchange.

[alexdimes]

Hats down to @joyce for keeping the development updates in check and being thorough with the process!

Wanted to quickly share that we’ve opened a dedicated Element room featuring a working group for teams ready to upgrade, and offer support (or discuss issues) from various ecosystem fronts (Parity, Parachain teams, etc.)

All information can be found in the separate Forum post linked above!