The Road to JAM

Thanks guys for your opinions, the discussion was fun. You guys understand more than I do. Is the gpu data below realistic? Because if so, it would be more feasible in the short term.

If you want to know more abou WonderZK you can go here:

October Updates

PolkaJAM:

Work continues on the 0.8.1 release of the Gray Paper. This release will include a rework of the auditing tranche mechanism, tweaks to the GRANDPA protocol, the final pieces of the PVM gas model, and a variety of smaller fixes addressing in particular external audit findings.

PolkaJAM now implements version 0.8 of the Gray Paper, with support for dynamically-sized validator sets and an improved gas model. The conformance fuzzer has also been updated to version 0.8. Work towards 0.8.1 has already started.

A proof of concept JAM light client built on smoldot is up and running in the browser (WASM), with a working browser demo.

A first version of the Zombienet integration for spawning JAM networks works locally.

Our stress-testing network scaled from 78 to 255 validators, with typical runs showing no errors over 24h. This network runs on the JAM toaster, with multiple validators per machine.

Minimal Relay (AHM v2)

As we’re completing the code, we want to prepare you for the Asset Hub Migration, you can find more details about the migration here.

As preparation, we want to invite you to an AMA session on Oct 23rd, 2PM GMT via Google Meets, you can add a copy to your calendar through this link. Bring any questions or concerns you have regarding the migration, we’re happy to address them there.

After addressing your concerns, we will also provide you a guide and an FAQ before the migration. Furthermore, please join our TG support chat to raise any questions or problems you encounter during or after the migration.

Offchain messaging:

Implementation of the Offchain Messaging protocol will be done in collaboration with SnowFork.

The scope was split into clear phases. The first version of the MVP can be found here. As the name suggests, offchain messaging will move the sending of the actual message content offchain. So, instead of routing all messages through the relay chain, like it is done right now, messages will be transported using a special networking protocol.

Parachain Services:

The first proof of concept of the Parachain Service now works on JAM. The PoC contains, among some smaller changes under the hood to make Parachains work in JAM, the following adaptations:

  • A working implementation of runtime upgrades. The major difference to the current relay chain based Parachains is that the runtime code is stored offchain from the PoV of the Parachain. The switch to this model is required because the runtime code compiled to PVM is otherwise too big.

  • The collation logic is adapted to work in the JAM context.

  • Resubmission of Parachain blocks to JAM in case of any problems. This reduces the number of forks, on top of JAM’s pre-existing “forkless” block production.

The team started turning the prototype into production code. If you would like to contribute to the project, feel free to pick up an issue with the label PoC2Production and help us review the code.

Best,
Parity Technologies

Thanks @joyce I’m curious if you are able to share what are the specs of each machine on the toaster and how many validators are running in them during the tests?

@florentina57 The general specs are:

CPU: AMD Ryzen Threadripper PRO 7995WX 96-Cores

Memory: 256Gig

Disks: 1.5TB NVMe

Networking: 10Gig

As @joyce mentioned the most recent tests have been running with 255 validators.

Is this enough info?

Thank you @ddorgan so the tests have been with 6 validators per machine?

@florentina57 with four validators per machine.

The ‘Add to Calendar’ link for the AMA doesn’t work for me. Is anyone else having trouble with that?

See if this link works - it’s only for google calendars though.

November Updates

PolkaJAM:

Most Gray Paper 0.8.1 changes are now implemented in PolkaJAM, including:

  • a dedicated compile hostcall for inner PVM modules

  • bounded preimage requests

  • the new Thin VRF scheme

Please note that these changes will not be merged to the main branch and available in nightly builds until GP 0.8.1 has been released and all changes have been implemented. The GP 0.8.1 release should happen within the next month.

A series of fixes landed:

  • integer overflow protections, with new work-report limits and test vectors (some of these changes are strictly part of GP 0.8.1 – see Bound work-report values by work-package limits by davxy · Pull Request #563 · gavofyork/graypaper · GitHub – but have been merged to the main PolkaJAM branch already)

  • fixes to disputes, authorizer pool updates and ticket encoding

  • fixes to hostcall behaviour so it matches the spec

  • PolkaJAM now avoids building on rejected blocks

  • a memory leak in the auditing subsystem was fixed

  • guarantees are now properly filtered when blocks are built

  • PolkaJAM will now avoid guaranteeing two work-packages which consume the same authorizer

  • nodes will now download block history after a warp sync

The new auditing tranche mechanism was designed, implemented and opened for review. For details, see the Gray Paper PR: WiP Proposal for rework of auditing tranches by zdave-parity · Pull Request #552 · gavofyork/graypaper · GitHub

All conformance repositories moved to a dedicated GitHub organisation, jam-conformance.

A new jam-crypto repository provides test vectors for Bandersnatch, Ed25519 and Reed-Solomon. It includes a Python reference implementation compatible with reed-solomon-simd.

CoreVM and the JAM tooling will be moving to a public repository; watch this space!

CoreVM’s outer-VM gas cost model was merged. New tests check that the simulator and the node agree.

CoreVM video codec optimisations, with no loss in output quality:

Before After
Encoded gas ~70M ~14M
Decoded gas ~64M ~12M
Encoded size ~2.2MB ~1.5MB

Minimal Relay (AHM v2):

  • Migration framework: the migrator state machine and shared types are merged. The lockdown stage is merged, and the account migration stage is merged.

  • On-demand coretime: the new on-demand pallet on the Coretime chain is merged. The final piece is in review.

  • Parachain registration: parachain deregistration will run through the Coretime chain after migration. Force registration is ready to merge, and only a few registration flows remain.

  • HRMP channels: this workstream sped up noticeably. The HRMP pallet was refactored so channel deposits can be managed remotely by the Coretime chain. The async deposit provider is ready to merge, and PRs for the full HRMP implementation are open.

  • FAQ: We’ve created a FAQ answering questions that came in after the AMA. Here’s also a short presentation introducing the project and its scope. If you have any more questions, feel free to leave a message on the support chat or here on this thread.

Offchain messaging:

  • Design v0.6 published (#13622). It includes fixes for issues found while reviewing the primitives. One of them is an edge case where a message channel could stall permanently, which was caught and resolved at design stage.

  • Core primitives nearly merged (#12676). They went through further review with SnowFork and are one approval away.

  • Supporting pieces:

    • MMR ancestry-proof helpers were merged and released in the MMR crate.

    • New fuzzers cover the MMR primitives.

    • The design for the minimal Parachain Service features was updated to lower gas usage for message settlement.

  • JAM side: a proof of concept was published for discovering bootnodes over a DHT on JAM (#13523). Work has started on running a JAM testnet with a collator prototype.

Parachain Services:

Specifications:

  • A storage proof format for JAM was proposed (JIP-16), so parachains can get proofs of JAM state.

  • Proposals were opened so non-validator nodes can recover work packages (JIP-22 plus a networking spec change).

Design refinement: implementing the spec in Rust surfaced small issues, which were cleaned up in the design. The design for work-package resubmission was agreed.

Verification: the formal Quint specification is being fuzzed against the Rust implementation, to check they behave the same.

Upstreaming: the authorizer is being upstreamed into the Polkadot SDK (#13453).