Deterministic PVF executor

[dmitry.sinyavin]

@koute

So do you propose us getting rid of wasmtime completely for PVFs and always use a custom VM?

Ideally, yes, but some mixed usage is also considered.

@koute

Is your prototype sandboxed in any way? Do you have bounds checking?

I’m yet to add red zones for linear memory and stack. Otherwise, we’re considering sandboxing as an external task wrt executor (see the corresponding project board). For sure, some sandboxing may be integrated into this implementation as well.

@koute

Are we fine with the performance hit that this will bring? How much of a performance hit would be acceptable?

I remember Jeff saying that, ideally, we’d like to have a 0.5-sec backing timeout, and right now, we live with 2 sec. So, the performance fits well into the current one but is right on the border of the ideal one. Further performance improvement would fit it into the ideal timeout, although I agree that’s not an easy task. I played around with Wasmer single-pass, it looks good, but it moves the other direction, that is, they’re solving an issue completely different from ours, with a focus on performance much more than on determinism.

@koute

If you’re going to rewrite this in a production-ready fashion it probably doesn’t make sense to start completely from scratch. You could reuse a lot of what I’ve done for PolkaVM - e.g. my assembler crate is very nice and is essentially production ready right now, and the sandboxing is mostly too.

Thanks, I’ll have a look! I thought about using iced-x86 for code generation but ended up just emitting raw bytes (and I’m yet to convince myself that there’s something wrong with that approach; I love its simplicity). But some parts of code, sophisticated enough, like function call sequence, would definitely benefit from using an assembler layer.

@koute

You shouldn’t underestimate the effort required to write a new production ready VM. (: I wrote my proof-of-concept of PolkaVM which could run real-world programs in two days, but getting it into a production-ready state is taking significantly longer.

Oh yes, I understand I’m not releasing it in a month. I though about step-by-step integration, first gated by an experimental feature, like we did with rocksdb/paritydb.

@koute

One idea I was thinking about (but haven’t experimented at all with) was maybe try to “recompile” WASM programs into PolkaVM programs and run them under PolkaVM, as a way of being able to transparently migrate everything we currently have on chain.

That’s interesting and should be evaluated. One concern is that the compiler performing conversion between stack-machine and register-machine architectures is the exact point where the determinism is lost, that’s why I decided to implement it as an almost pure stack machine in the first place. But I’m not saying it’s impossible, it requires deeper research.

@koute

Any stack depth limit we have should be completely independent of the underlying CPU architecture. So I think the way to handle it is to 1) have a logical stack depth limit that we know can use at most X bytes, and 2) always have the native stack size be at least X bytes big so that we know we will never hit it.

@koute

Couldn’t we limit the logical stack depth of the PVFs by just instrumenting them, and run them on wasmtime as always? We could inject the relevant stack metering code into the WASM blob, and get the code to deterministically trap if it’s exceeded. Have we tried this already?

What you’re describing is the exact approach we’re using right now! Our stack limiter is used for PVFs. But it only measures the value stack (which, in the native code execution stage, is a pure abstraction in the case of Wasmtime), and the machine stack is limited by the execution worker. Because of the non-deterministic nature of Wasmtime, the call stack limitation cannot be deterministic (it will overflow in different execution points on different versions of Wasmtime and different platforms).

What I tried to implement is to get rid of the necessity to have a logical limit at all. In my case, the native stack is a mixed value/call stack, and the value stack is not abstracted, those values really reside on the machine stack. Thus, its overflow is always deterministic. The only limitation imposed is the platform bitness. It only works on 64-bit platforms, but within that bitness, it should be fully deterministic.

@koute

I also tried to grab your PoC code and run my “standard” benchmark with it to evaluate performance compared to everything else (as I’m quite curious how fast a naive WASM VM would be), but alas it looks like you’re still missing some opcodes. (:

Yes, it doesn’t support all the opcodes yet. It has some historical background At the beginning of the development, I (as well as other teammates) was sure that using floating point is forbidden in parachain runtimes, as it is in relay chain runtimes, so the idea was to only implement a subset of Wasm that is needed for parachains. But when I started experimenting with real parachain runtimes, I found out that’s not the case, and parachains are still using a small subset of FP operations brought in by their library dependencies (you may probably remember my forum post on that). After that finding, I had to implement FP operations but only did that for that small subset required by parachain runtimes. I will support all of them, of course, I just don’t need them right now for experiments and testing. (NB: if/else construction is not supported yet either. It’s never generated by LLVM compilers, they always do conditional branching instead, so I postponed it as well).