The possibility of distributing multiple separate binaries to validators has already been touched on at UX implications of PVF executor environment versioning. But that thread kind of went off-topic, and we have new requirements now due to recent security work. So we (me, @dmitry.sinyavin, and @eskimor) want to restart the conversation from a different angle (security + future-proofing).
The immediate motivator for this change is splitting the PVF worker binaries out of
polkadot. There are several reasons we want to do that, mostly having to do with security:
- We can statically link the worker binaries with musl. This lets us better sandbox the expected syscalls as they do not depend on the system libc, plus it’s a win for determinism.
- @koute wrote a syscall detection script which to my knowledge only works with separate binaries.
- Having smaller binaries to run untrusted code helps protect against possible ROP gadget attacks.
- It may eventually be needed for virtualization.
- We are considering using a global allocator wrapper which @dmitry.sinyavin wants to enable on only the workers, without suffering the performance hit on the rest of
(Side note: work on multiple binaries has already started, but what we already tried was splitting out the binaries while preserving the existing single-binary UX. The idea was: build the binaries during
polkadot compilation using something similar to
wasm-builder, embed them in the
polkadot binary, and extract the binaries at runtime. In addition to being rather hacky/unorthodox, multiple issues with this were discovered with this during development. One is that there’s no good way to do the extraction step on MacOS.)
So due to this security work, we want to start distributing multiple binaries soon. One way is with a simple zip file. But, since we already need to change the validator UX, we should do it properly. We should either provide an installer binary/script and/or distribute through package managers. There are several benefits to this:
- It is future-proof, as we wouldn’t have to change the validator UX again whenever new requirements surface later.
- We could do additional configuration that requires root.
a. For example, as part of securing the workers @eskimor suggested setting up a new separate user that would run the PVF binaries. [I still need to research and write the issue for this.]
b. Also, I believe @dmitry.sinyavin mentioned that another feature we are eyeing,
cgroups, requires some configuration.
- We can do some security checks on the generated binaries with e.g. this Perl script.
- Installers and package managers are common practice, and the UX is arguably more familiar (and simpler) than manually managing multiple binaries from a zip file.
Some possible downsides are:
- Validators may not trust package repos to be secure – they can always inspect and run the install script themselves.
- It would be more work on our side to support multiple package managers – probably good to bite the bullet and do this sooner rather than later.
- Others? Let’s discuss!