Virtual Operating System for JAM - program with wink! and zink!

JAM development is progressing smoothly, teams are delivering milestones and sooner than later the first JAM services(like parachains service) will come to life in the test environment. It’s great that people will be able seamlessly migrate their Substrate parachains to a more capable home but this also opens up a door to new opportunities.

When first hearing about JAM I was skeptic to the point of annoyed, not doubting the technology which is always great, but thinking, damn we did it again … introduce a new complex concept that most people won’t get when we haven’t even exploited what we currently have, now all attention and resources are going to flow into the new thing neglecting our current users and needs(e.g. UX/DX). Parity’s response to this problem came in the form of the Polkadot Hub, is great as it makes Polkadot core offerings familiar to the more abundant EVM developers :+1: But what if we could attract even more non-blockchain savvy keyboards to our ecosystem?

WASI and VOS

The WebAssembly System Interface allows WebAssembly modules to interact with system resources like files and networking, enabling them to run outside of web browsers in various environments securely. As a Rust developer for example I can take a regular program(no specific domain knowledge required) and simply compile it with --target wasm32-wasip2 to get a cross platform binary that “just works” across common operating systems.

use std::fs::File;
use std::io::{self, Write};

fn main() -> io::Result<()> {
    let mut file = File::create("output.txt")?;
    writeln!(file, "Hello, World!")?;
    Ok(())
}

This is different from other WASM programs(like Substrate runtimes) that usually define a custom “contract” with their host environment and require specialized tooling and knowledge of said interfaces, instead of the well known “system calls” used by WASI that a standards following runtime will already implement.
WASI is also transitioning into a component model that allows extending the basic capabilities of WASI with new sets of higher level interfaces and great cross-language interoperability, e.g. a Python program can easily use a function from a Go library that uses the wasi-nn interface to accelerate ML inference loads.

WASI, specially the idea of an environment centered around it interested me from the start and I’ve been looking for ways to translate its benefits, specially the better developer experience, to our ecosystem. I didn’t see a good way for WASI to fit within Substrate but JAM brings new possibilities!
We can have a layer that abstracts the assumptions of an “std program”(e.g. a file system, a long running process) to let developers continue creating the same kind of programs they already know, said layer is a lightweight Virtual OS that in the case of JAM would hide away the complexities of the new system(e.g. the data availability layer can be exposed to apps as a file system).

:information_source: The V in VOS also stands for Virto :wink:

Some of VOS properties:

  • Developed as en embedded system with the async runtime embassy, it makes no assumptions on where it’s going to run, could be the browser, a linux server, directly on metal(e.g. a micro controller) or a virtual machine like PolkaVM.
  • Wasmtime is used as the go-to WASM runtime that recently added support for no_std environments and custom targets with the requirement of pre-compiling programs to the target architecture, this is actually a plus for the JAM use case as we can deploy WASI programs already compiled to PVM.
  • Besides the WASM runtime Nushell is provided as a shell scripting engine to allow easy composition of programs and creation of powerful data processing pipelines(support for JAM and embedded might take a while).
  • By default the OS talks to WASI programs with a simple to implement protocol over standard input/output, it’s actually Nushell’s plugin protocol which is nice for development and testing of programs if you have nu as your shell.
  • “System ports” are well-known system services that provide means to send commands and scripts to the OS to be run asynchronously. e.g. call a function in a program with a simple POST HTTP request or connect via SSH interactively to explore your installation and try scripts.
  • It can be used as a back-end framework or even as a decentralized “back-end in the front-end” so the same HTTP calls you make to a server can be processed locally on your device(in a web worker).
  • Different “user space” virtual file systems(similar to FUSE) can be implemented to offer different capabilities depending on the target platform. e.g. an abstracted DA layer. I’ll also explore implementing a Matrix file system to store a program’s state in encrypted Matrix rooms.
  • The OS adopts a very asynchronous architecture expecting guest programs to also be async and yield control back to the host regularly, although embassy allows for some level of preemption with interrupts.
  • A VOS instance is called an “agent”, the OS internal execution loop is modeled like a robot’s, sensors bring data, a planner makes decisions to tell actuators to “do stuff”.
  • Performance and memory consumption are priorities but convenience and easy of use for the developer should always be a bigger priority.
  • To provide a well integrated and familiar experience within out ecosystem, a couple of ink! dialects are used to create most programs.

wink!

Although VOS should run arbitrary std programs, it’s nice to provide a familiar default developer experience. ink! is used as inspiration, it has a great design with a simple and elegant interface, so it’s worth creating a similar dialect for our WASI programs.
The wink! macro is a simple one that produces the boiler-plate necessary to setup the required async runtime, de/serialize the program state using the file system, do processing of incoming messages via standard input and dispatching calls to the right handlers.
A wink! program should look and feel like an ink! contract :wink:

#[wink::bin]
mod demo {
    #[wink(storage)]
    #[derive(Default)]
    pub struct Demo {
        counts: Map<String, usize>,
    }

    impl Demo {
        #[wink(message)]
        pub fn count(&mut self, who: String) -> String {
            let count = self.counts.get(&who).copied().unwrap_or_default() + 1;
            self.counts.insert(who, count);
            format!("called {} time(s)", count)
        }
    }
}

But remember we are inside an async WASI environment! so you’ll have more APIs and tools at your disposal, like defining long running async functions that produce streams of data processed downstream, easily call other programs or scripts, import dynamic libraries(wasi components) that expose OS functionality and probably more.

Depending on the used features and hardware capabilities some programs will be available only on specific platforms(e.g. painting to a screen might not make sense on JAM) but expect many programs to run the same on your watch, your browser or your company’s cloud.

:information_source: Virto will provide commonly used programs like chain querying/tx submission, or zk proof creation/verification.

zink!

VOS agents won’t live in isolation, they’ll want to interact with each other! Just like an agent can talk to known “friends” over trusted channels, oftentimes they’ll want to interact with other unknown agents or systems in trustless environments(like JAM) where proving each other facts without reveling sensitive information will be crucial. e.g. Your organization’s agent might want to prove a public government taxing agent that it’s still a small org without revealing information about its employees.

:information_source: Virto has many more use-cases for organizations where we want to maximize privacy for individuals within organizations but maximize transparency of the overall organization so they can be compliant with other protocols.

fn main(x : Field, y : pub Field) {
    assert(x != y);
}

The default mechanism for creating privacy preserving zero knowledge proofs in VOS will be with the popular zk-SNARKs. zink! programs use Noir, one of the best and simplest languages to create zk programs/circuits, it’s Rust-like syntax mixed with the ink!-like structure provided by zink! offers a great and familiar experience to developers of our ecosystem that won’t need to know much about low level details of zero-knowledge cryptography.

zink! proofs will be easy to generate from wink! programs which provide the necessary private and public context. Verification should be equally easy from other wink! programs or ink! contracts on a Substrate chain.

Closing notes

VOS is our bet to create a great unified developer experience for decentralized and hybrid applications, it’s not limited to become a JAM service which is great as it can attract non-blockchain developers and offer a uniform experience across multiple platforms and environments like hardware, front-ends, traditional server applications and more.
We are in early stages of development so its design might change, but expect a soon to be released MVP showcasing its usage for back-end APIs and as in-browser backend that can connect to a Substrate blockchain.

:information_source: Virto will ask funds in Kusama to expedite VOS+wink!+zink! development. This post serves as extended context for the proposal.

1 Like

@olanod .
Is there any place where we can test this new code?