UX implications of PVF executor environment versioning

Tech Talk
1

PVF executor environment versioning for dummies

To execute a PVF, we’re currently using Wasmtime, which is a compiler and a runtime in one bottle. When a new version of Wasmtime is released, we update the Wasmtime version in our node implementation. Unfortunately, that is not a deterministic behavior: if a PVF is successfully executed with Wasmtime version N, nobody guarantees it can be successfully executed with Wasmtime version N+1. At the moment, we’re lucky. Nothing broke up horribly because of those upgrades. But we cannot just hope to be infinitely fortunate, so we want to introduce PVF executor environment versioning. That is, we want to carry several versions of Wasmtime along with the Polkadot node and to be able to execute a candidate produced in some session with the exact version of Wasmtime that was effective in that session. That should make things deterministic and infallible.

Github issue for those who want to go beyond tl;dr: Explicit versioning of PVF execution environment · Issue #4212 · paritytech/polkadot · GitHub

The problem with the UX

I always considered the versioning issue a purely technical problem, but in a discussion with @rphmeier, some UX concerns were brought up. If we want to carry several versions of Wasmtime with our Polkadot node implementation, it basically means “several files”. Not talking right now about whether they would be binaries of dynamically-linked libraries, but generally, the Polkadot node ceases to be a single-binary distribution. So we should carry and manage some auxiliary files to make the versioning happen. That would affect the node operator’s user experience significantly. So I’m creating this topic to discuss the implications and possible ways to handle them.

Possible roadmaps

From my perspective, there are several ways to proceed:

  1. Create a distribution as an archive with several files inside. They may reside in a single directory or a directory tree. Documentation should contain obligations for the node owner to preserve the file structure/directory tree. It’s the easiest choice for us but the most annoying for node owners.
  2. Create a single-binary installer that properly lays out the needed structure and upgrades an old version to a new one. The most common way in the industry, still prone to errors due to space management procedures, etc.
  3. Put some effort into keeping the distribution single-binary. That is technically possible. ELF binaries may contain sections not loaded by default; we can load them on demand and link against them in runtime, they may represent different executor environment versions, and everyone would be happy. But that requires some currently unknown amount of work. It’s the best choice for node owners but the most demanding for us.

What’s next?

Discuss!

2

Not sure why you come here up with ELF binaries and linking at runtime? Sounds like you over-complicating it.

With dependency renaming I would say that this is the most simplest solution that we could implement. No need of handling different binaries or whatever, which also requires us writing code etc.

1 Like
3

Yeah, I didn’t know about that feature, thanks for pointing it out! That really makes keeping everything in a single binary much easier.

The only question is if we definitely want to keep everything in a single binary. We’ve been talking about separating binaries for quite some time now; the release binary is over 200+ Mb now, and the debug one is a 4 Gb monster. And every time worker is spawned, we spawn it from that binary. Now, there are voices that we’d like to increase the number of parallel workers, and imagine each of them is carrying a payload of 5 different versions of Wasmtime. That makes some scary numbers.

I can imagine different scenarios for keeping a single-binary distribution while using smaller-binary workers (as a dumb but straightforward example, a glued binary of the main binary and smaller worker binaries, and the main binary writes smaller ones to disk on startup), but that should be evaluated from the UX angle of view.

4

Why is it not guaranteed, by curiosity? Wasm is governed by a specification, and the non-deterministic aspects like the stack limit are handled as well as far as I know.

The issue you linked talks about differences in the list of host functions, layout of the structs, etc. but none of this is related to wasmtime.

What the advantage of separate binaries? Linux is normally smart enough to avoid loading the same binary into memory if you execute multiple times. In fact it shouldn’t even load the parts of the binary that you don’t execute.

5

I agree, and I’ll insist that we don’t bother with multiple binaries. We should just compile multiple versions of wasmtime as normal cargo dependencies inside a single binary and just be done with it.

Note that from a practical point-of-view this might not always be straightforward to do. IIRC wasmtime has some dependencies which use non-Rust libraries (IIRC it was zstd I think? And maybe one more? I don’t remember.), and those deps must be unique within the whole crate graph or you’ll get a linker error. So if wasmtime 20 uses dependency foobar 1.0 and wasmtime 21 uses dependency foobar 2.0 then both of those wasmtimes cannot be included in the same program.

Of course this can be trivially worked around by just us temporarily forking the older version of wasmtime and updating the dependency. So a little extra work, but not a big deal.

There’s one more issue here that I think should at least be explicitly mentioned: in theory some versions of wasmtime could be found vulnerable to critical security issues (like one here Guest-controlled out-of-bounds read/write on x86_64 · Advisory · bytecodealliance/wasmtime · GitHub), so if our model is going to be to keep the old versions of wasmtime around what are we going to do in such a situation? Also, what about dormant unreported security issues in old wasmtime versions? We should probably at least have a mechanism that would disallow a new parachain/parathread from requesting an old wasmtime version if there’s a newer one available.

Things like e.g. different amounts of native stack being used (although we have some mitigations in place so that at least we don’t overflow it), different amount of time it takes to compile a program, and of course different amount of time the compiled program is going to run (of course this problem is inherently unavoidable since everyone’s running on a different hardware, but I could imagine wasmtime suddenly improving their optimizer where the execution time could get orders of magnitude different).

The objective here is, I suppose, not to make it completely deterministic (because that’s impossible), but just to minimize any extra source of nondeterminism as much as we can.

2 Likes
6

Imagine a function that is when compiled with version N uses 2040 bytes of native stack, and when compiled with version N+1 it uses 2056 bytes, and our limit is 2048 bytes. Such changes in Wasmtime already took place before.

7

I agree that’s the most straightforward way; we just need to figure out how to manage that stuff. Let’s say, version 1 of node software links version 1 of Wasmtime, node version 2 links versions 1 and 2 of Wasmtime, and node version 3 drops support for Wasmtime version 1, keeps support for version 2, and introduces support for version 3. Now, the network consists of nodes of all three versions. Executor environment parameters keep some indication of which Wasmtime version should be used in which session. What if node version 1 is requested to validate a candidate with Wasmtime version 3? Is it just a no-show? But what if we still have a lot of nodes of version 1? The problem is obviously solvable, but the solution needs to be carefully designed, that’s the point.

Also, as updating the Wasmtime version is a part of the usual workflow, it should be automated somehow. Right now you’re bumping the version manually, but if we have more versions in parallel, I don’t think you’d like to manage that zoo by hand.

1 Like
8

But the stack limit is supposed to be covered by: https://github.com/paritytech/substrate/blob/86c6bb9614c437b63f3dbd2afddef52f32af7866/client/executor/wasmtime/src/runtime.rs#L395-L419

In particular:

Therefore, it’s expected that native_stack_max is greatly overestimated and thus never reached in practice.

Sure, the stack usage of a function could suddenly jump from 2048 to 16384 bytes, and overpass the overestimated native limit, but that would surely be considered a wasmtime bug.

9

In general, I don’t understand why this approach is taken.

The entire point of a blockchain is to be governed by a very precise specification that every node closely follows.

If you move towards embedding multiple versions of wasmtime “just in case”, you are going in the opposite direction. Instead of having a specification that implementations follow, you say “the specification consists in what this specific implementation [wasmtime] accepts to run with this specific stack limit”.

It essentially becomes impossible to implement Polkadot without depending on specific versions of wasmtime.

The fact that you have to be able to very precisely define which PVFs are accepted and which aren’t is a constraint that must fundamentally be followed, otherwise we don’t have a blockchain anymore.

2 Likes
10

I proved that statement wrong some time ago :frowning:
The estimation relied on (and still relies on) the hypothesis that the native stack size is proportional to WASM value stack size. That’s not the case.

It cannot be considered a Wasmtime bug as Wastime does not ruin the spec here. Spec tells nothing about the limit itself and about how much native stack should the native code use, so they’re in their own right here, they may just change compilation logic between versions and the stack size may change dramatically, there’s nothing wrong with that beyond the fact it’s not acceptable in our case.

But we don’t want to be bound to a single version of Wasmtime, and even to Wasmtime itself, we want to follow Wasmtime updates and have the ability to replace the executor if needed! And we want to do it deterministically. If we upgrade Wasmtime and some PVF that was successfully pre-checked and executed by the previous version ceases to function properly, that is where the determinism is ruined.

Exactly, and as we have a highly configurable blockchain here, we don’t want those constraints to be written in stone, we want them to be deterministically changeable on a per-session basis. What was acceptable in session 100 may become unacceptable in session 101, but a candidate produced in session 100 should still obey the rules of session 100 and shouldn’t get disputed in version 101 retrospectively because the rules have changed. That’s what we’re trying to achieve here.

3 Likes
11

The problem with parachain consensus and PVF execution is that we have to deal with untrusted runtimes. This is an order of magnitude harder than handling determinism for let’s say the relay chain runtime - which is trusted.

Think of it this way: We have to assume that there are people who search for even the smallest difference and exploit it. It is in my point of view unrealistic for such a complex piece of software to be solely defined by a specification and expect it to be robust against that threat model. It is already hard, if we only assume one implementation! Even if the spec was perfect, implementations must not have a single bug!

Also we are fine with only having a single implementation of the runtime for Polkadot. Having multiple would be a recipe for consensus problems. It is open source, and it can be changed and altered as we see fit. Where we need specifications, is places where we have to interface with other implementations - e.g. light clients. But relying on specifications in consensus directly is a recipe I don’t see working out realistically ever.

For wasmtime in the case of PVF we have to treat it as part of consensus - we can swap it, but deterministically. Having different implementations used by validators at the same time, is a challenge I don’t see us prepared for.

Different story is, if we added some form of light client for parachain consensus: This is outside of consensus, so here the damage a malicious PVF can do is limited. What I can see us being able to do, is spec it the best we can, then other nodes and light clients can use a different implementation e.g. one that works better in the browser or whatever - if things break there, you just display an error to the user, but you won’t raise a dispute.

For validators, I think the most secure route is to have them all execute the same code, just as with the runtime.

2 Likes
12

I think we have to expect and be prepared for such bugs in our dependencies, and do what we can to run deterministically regardless. If disputes happen, causing slashing and economic impact, it would be pretty lame to say “not our fault” and punt responsibility to wasmtime – especially since such bugs already happened before.

1 Like
13

This will be wrong in the long term and I also already highlighted this multiple times by now. The overall goal will need to be conformance testing. Yes we will need to prepare all these things, but the assumption that all nodes will run the same code is nothing we should assume or even think about. When we hit such a bug, we will need to check what happened, reproduce and then use Polkadot governance to revert slashes.

There is already the first fundamental implementation of conformance testing on the way that will lay the foundation to help preventing these issues: Parachain Validation Conformance Testing | Polkassembly

I think another easy option would be to record all PoVs to re-execute them when there is a wasmtime upgrade. This can be done very easily and will help us to discover bugs before we try to deploy on the network.

14

Exactly this! This is exactly like Ethereum that also doesn’t force every body to use the same solidity vm to be part of consensus.

2 Likes
15

This does not work. We are worried about consensus being broken on purpose! Testing a new wasmtime with existing PVFs before deploying/releasing is definitely something we should do and will prevent us from borking the full network with a buggy compiler, but it does not resolve the issues we are worried about: We are worried about edge cases, edge cases that are usually totally irrelevant to any real world PVF, but can be exploited by an attacker to trigger disputes, getting honest validators disabled and slashed. It does not help to test existing PVFs. The attacker will wait for deploying his malicious PVF until the buggy version is deployed. He won’t reveal it before.

Ethereum consensus is a totally different beast and way simpler than ours. Point being, we are working hard on making unjustified disputes as unlikely as possible. You can see that this is not an easy task, by the fact that we are still seeing disputes on Kusama and this is even without having on-demand parachains nor an actual attack!

What is important to understand is the angle we are coming from: Also from an ideal, similarily to “everything should be speced”: We are coming from the ideal: “There should be no disputes in production!” and this is for a reason:

For actual malicious nodes, the threat of disputes and slashing’s existence should be enough for them to not even try. What we want even less is getting honest nodes slashed because of bugs. An absolute no-go: Having outsiders trigger those disputes which get honest nodes slashed!

Yes we can revert slashes, but consequences go deeper: The design of disputes assumes that we can guarantee liveness (and scalability), by slashing validators disputing valid candidates. This falls apart, when disputes can be triggered by outsiders having nothing at stake! We could harden the system further by e.g. disabling a parachain when too many disputes are raised against it, but this firstly becomes moot with on-demand parachains - there can be lots of them! Secondly, it can backfire: Us DoSing a perfectly good parachain because of some validator going rogue.

In a nutshell, if it were possible by an outsider to trigger disputes on the network, we either have a security problem or a liveness problem or both! Hence we try our very best to make this as unlikely as realistically possible. Let’s just spec it and have different implementations on validators is an absolute nightmare from this perspective!

I do understand the argument: We don’t want “whatever is accepted by wasmtime” to be our spec, but that is also only the case if you are very pedantic. The spec is “wasm”. Any compiler that faithfully translates wasm to machine code should do in practice … for honest code. There are corner cases though that can be exploited by malicious code, which is why we have to be very pedantic on validators. For actual honest application code “wasm” as spec should be sufficient, in the sense that if you are an honest parachain developer testing your code with a different compiler, it will with very high probability also be accepted by Polkadot.

2 Likes
16

To better understand the problem. Let’s assume we have two different implementations.

First scenario: > 1/3 of the validators use one and >1/3 use the other. Now if there is some flaw in one of them an attacker can make create a dispute that does not even conclude! This is a liveness nightmare. Nobody will get slashed, so the attacker can keep doing that indefinitely. Expect a complete DoS!

Second scenario: 2/3 - 1/3 split:

We slash the 1/3. If we choose to also disable them then the attacker just managed to knock out one half of all the honest nodes (byzantine assumptions). Suddenly our byzantine assumptions are no longer preserved and the security of the network is no longer maintained.

Alternatively, we don’t disable them: Then we have the same situation as in the previous scenario.

2 Likes
17

Yeah for sure I know this and get this. However, the entire thing here about using a specific wasmtime version or doing an upgrade together will not help against these edge cases, because as you said an attacker would just wait until it can deploy the exploit.

Parts of the network rejecting to sync or starting to fork doesn’t sound like it isn’t being any problem :wink:

For sure! No one said that it would be easy to implement all of this or that there wouldn’t be any bugs.

Welcome to the nightmare :slight_smile: Just specing it will not solve it nor will make it easier. We need to accept that this will not be easy and will take quite a lot of time to fix it properly and also that we will encounter slashes because node implementations are not agreeing. That is nothing you will be able to prevent. The earlier we accept this, the earlier we can try to define the boundaries as good as possible and prepare when this kind of accident happens.

Yeah, I mean no one is argumenting against removing slashing. But you could also run into this issue by having the same node version running on the entire network on different architectures. There being a bug for one architecture and not the other. Or different OS versions. There are probably endless ways why the same implementation can behave differently.

1 Like
18

Sorry but that’s IMO just wrong. If a piece of software is actually so complex that it can’t be specced (which is not the case here IMO), then the only solution to me is to simplify it, not accept this reality.

I’m sure that they consider some things as bugs even if it technically conforms to the spec. If running a basic balance transfer started taking 10 seconds and 50 GiB of memory, surely they would fix it.

Let’s say wasmtime was a Parity project written by Parity devs. Would you still decide to embed multiple different versions of wasmtime in Polkadot, just in case they have accidentally introduced a bug in their own code?

If yes, why don’t we do that for other critical pieces of code such as the host functions implementations, ParityDb, or the syncing algorithm?

If no, then why this double standard? It’s not like wasmtime is closed source. Parity or Polkadot devs can follow what they are doing. I don’t see why we should trust the wasmtime devs less than Parity devs.

I don’t understand why wasmtime is particularly treated as an arcane magic piece of code that we have to write defensive code against, rather than a piece of code that can have bugs that can be fixed like is done for the rest of the codebase.

If Polkadot still exists in 10 years, I strongly believe that the only way for it to strive and not become a dummy network that nobody uses is to eventually phase out Substrate and its low quality code with a new different implementation. If the decision is made that validators can only ever use the official Polkadot client, I think would effectively kill Polkadot.

1 Like
19

They will not if the difference is not as major as in your example and only affects blockchain guys. They’re not targeting blockchain projects. Afaik we were already trying to push some changes to Wasmtime that would make our life easier and failed in many cases.

Definitely no. It was my point in the previous round of this discussion with @bkchr: if we control either the VM spec or VM implementation, we do not need versioning in this form, or we can implement it in a more natural way, not linking several versions. But we control neither VM spec nor VM implementation, that’s why versioning is needed, and that’s why it’s different from Solidity VM case.

Because 1) Wasmtime behavior directly affects parachain consensus and 2) Wasmtime devs don’t care if their changes affect parachain consensus.

Because Wasmtime is an arcane magic piece of code :man_shrugging: What looks like a bug from our side (e.g., increasing native stack usage in a new version) isn’t a bug for Wasmtime guys. They won’t fix it, and they would block our attempts to revert their changes. And at the point where such breaking change is introduced, if we don’t have versioning, our only choice is to fork Wasmtime (that was also discussed, I hope not very seriously).

In a long run, I believe (and it’s just my personal opinion) we’ll have to give up executing PVFs with Wasmtime anyway. We’ll have to either take control over VM spec (that is, to develop our own VM; I don’t think it’s a good way to go) or take control over the implementation (that is, to create a deterministic WASM executor which sounds more realistic to me) or to use another VM that has a more deterministic spec initially (RISC-V probably; didn’t dig deeply yet). I don’t see other ways to safely execute untrusted code. And when we’re ready to switch, executor versioning will be of great help in that process. The other option is to keep things as they are and pray that nobody will ever want to break things.

2 Likes
20

BTW, the entire assumption that wasmtime is the only wasm vm being used to validate PVFs is not true anymore: Release KAGOME v0.9.0 · soramitsu/kagome · GitHub