Dynamic Backing Groups: Preparing Cores for Agile Scheduling

[sandreim]

@rphmeier

With an algorithm that assigns backers to parachains more intelligently and adaptively we get much closer to fully solving (3) and (5). It’s not perfect for (5) as Polkadot still sells fixed ratios of data and execution as coretime. However, the decoupling of backing groups from core indices is a useful step in the direction of granular markets here. i.e. we have no hard requirement that a “core” even corresponds to a particular amount of resources anymore.

I think the PoV size and execution time for the parachain blocks needs to be known in advance for any algorithm that wants to assign backers to parachains efficiently. It can also work, if the core time the parachains buy also has a constraint around the maximum execution limit to fractions of a single core. WDYT ?

@burdges

Any validator has more approvals assignments than backing jobs, so we’d wondered if validators could reject some of their approval checker assignments. Yet, we’ve never figured out how this impacts security, so probably a bad idea.

Instead, we now suggest overloaded nodes simply delay their own backing work, which brings no soundness consequence. It’s also a reason to have smaller backing groups, so the system feels the back pressure sooner, which should improve performance but harm liveness.

In case of disputes, we already backpressure by we dropping bitfields and backed candidates because we prioritize the dispute statements on chain vs anything else. So, we’d prioritize the PVF execution to work on dispute checks first, then approval checking and backing lastly.

@rphmeier

any candidate (or bundle) can go in any core as long as they are backed by enough backers from the parachains’ group. The relay-chain block author makes a selection based on what they have received from gossip and is rewarded for packing cores optimally.

How does this relate to bundling for approval checking? I expect the CandidateCommitments to include execution time for candidates and we could be even more efficient down the pipeline if we bundle things further as ExecutionGroups like in Idea: Core Groups · Issue #607 · paritytech/polkadot-sdk · GitHub