eBPF hack report
@Alex and I discovered that we both have been entertaining the idea of using eBPF for contracts instead of WebAssembly (wasm).
Disclaimer: Speaking for myself here, maybe @Alex can give his perspective below.
Intro
After working with wasm in various environments, we naturally gained a deeper understanding of wasm, it’s strengths and it’s costs.
It enables the safe and performant execution of portable code with easily achievable determinism. However, it is also more complicated than it could be if focusing on our scenario. Below are some examples that come to my mind.
Runtime Structure. The runtime structure of a wasm module is rather flexible. A wasm module can declare different entities: functions, memories, tables and globals. Those can be exported from a wasm module, or they could be also imported into, meaning that those items must be provided at instantiation time.
We don’t really need that flexibility and could get away with much less. E.g. modeling global variables as separate entities might improve compiled code quality, but for linear time compilers this does not really matter.
Interaction with host. In wasm the main way to interact with the host is through function imports. The module would specify functions, their names and signatures, and the host would provide implementations of those during the module instantiation.
Memories. Memories can declare an initial size and optional a maximum size. During runtime a memory can grow up to its maximum size.
I am pretty sure contracts can get away with a limited amount of memory. pallet-contracts used to limit the memory size at fixed 1 MiB. Solana contracts are enjoying 32 KiB heaps.
Bytecode. There are several things, so I will highlight on only a few.
Wasm bytecode is a stack machine. The stack is unbounded. Moreover, a function can declare arguments and locals. That all means you actually need to perform some register allocation to get decent performance.
Wasm bytecode operates on primitive values of I32, I64, F32, and F64 types. Contracts don’t really need floating points and if they need it badly, they can use software implementations. Also, I am pretty sure there is no need for two integer types, and having only 64-bit numbers is pretty much OK.
One of the most obscure corners of Wasm is the polymorphic stack. This is a mode that makes the operand stack switch to special rules. It is enabled when the execution is trivally unreachable. But since it’s unreachable, it does not really matter and could as well be eliminated.
All those features bear some cost, either in the complexity of the API, the complexity of the compiler implementation, or performance.
One of the reasons why Polkadot still does not embed the API for Wasm VM is its complexity.
Also, it’s easy to abuse typical optimizing compilers and adversaries can craft contracts doing that. A canonical solution for this is to employ a linear-time (in code size) compiler. At the same time, we want to have a fully optimizing compiler for the Substrate Runtime / PVF execution. That means that in one way or another, Runtimes and contracts are destined to use different VMs.
Those thoughts made me consider taking a look at the other options. That’s how the idea for the hackathon started.
Learnings from Hackathon
eBPF’s instruction set is register-based. eBPF defines 10 general-purpose registers. eBPF does not split code into functions. It’s just one big blanket of code. No args and no locals. That all makes it a good target for translation into machine code in a single pass. Check out the source code of eBPF to x86-64 from the Linux kernel.
A big obstacle was the lack of documentation. A lot of time was spent reading the code, issues, and doing the good old trial and error.
Vanilla eBPF is not a really welcoming environment to write contracts. Since this tech originated from the kernel, it limited the space of expressable control flow. Specifically, it limited the usage of loops. People would use C with special macros to specify bounds for each loop. That immediately constraints the usefulness of the whole thing.
To avoid this, we decided to use the tech from Solana. Namely, they forked rust/llvm and rbpf. The fact that eBPF requires usage of a custom rustc/LLVM backend kind of diminishes it’s value. Also, managing their custom toolchain wasn’t the best experience ever, but hey, at least we did not have to repeat all the work ourselves.
After getting all setup, we got our first artifact: an ELF file. ELF is a bit archaic but mostly fine. But how the interaction with the host is set up is a bit weird. I still haven’t figured it out completely, but the same instruction is used for calling the code within the module and the host (syscall).
From a glance, it seemed that the call instruction has a 4-byte immediate value to specify the call target. Usually, it seems to be an offset in the text section. However, syscalls seem to be specified with special magic cookie values. I guess this is not standard behavior, but registering a syscall in Solana’s RBPF VM would take a hash (murmur) from the syscall name, and I assume a call with that hash would be interpreted as a syscall. I can’t call that an elegant solution, but fine, that works.
It does not end here. Even though the hash values can be computed upfront, Solana’s runtime depended on dynamic relocations. That is, all syscalls would be emitted as
LABEL_001: call -1 # my_syscall
paired with a relocation entry that specifies that the particular my_syscall is called at LABEL_001. When the ELF is loaded prior to the execution, the loader would go over all relocations, compute hash, in this instance of my_syscall, then write fixup all values from -1 to the actual hash value.
To be fair, Solana folks do realize the problem since there seem to be some feature flags, that compute those hashes offline. But for now, it appears to be off by default.
Another quirk that we noticed about Solana toolchain is that it seems the recommended way is to produce shared object ELFs, i.e. .so files. Yet, despite being shared objects they have a single entry point, which is specified in the ELF header. The way how it works is unclear since a binary having a no_mangle function is automatically assumed to be an entrypoint by the forked rustc/LLVM. Solana’s RBPF VM also seems to make some assumptions about this.
Looking deeper into RBPF made me unhappy. The overall quality of code questionable. An unsoundness hole is looking back at you right from the primary user facing API. The memory model is also very questionable. Per their overview, their address space is 64 bit, is not linear (i.e. has gaps) yet really tiny amount of that memory is used. This requires them to emit a pretty complex bounds check before each actual memory access. RBPF does this by emitting a call into a non trivial Rust function that does bounds check. Once more, per one memory access they would do several more accesses while calling Rust and doing bunch of work. To give you an example, in-kernel eBPF elides all checks based on verification. Wasm typically catches out-of-bound memory accesses after the fact, being almost zero-cost.
On the bright side, the Substrate Runtime Interface for the eBPF turned out to be way simpler than the wasm counterpart. However, don’t be deceived, since the API in this case also includes strict promise on stability. Changing, and that includes fixing, even small things can lead to a disaster.
When it comes to the eBPF format itself, one unpleasant surprise was that each instruction is 8 bytes long. This might be good for decoding simplicity, but not really great size-wise. Although those would probably compress well.
Conclusions
I don’t really have a good grasp of all the little details of eBPF. It’s my first time interacting with it closely.
The first impression was not really great, but this is mostly because of Solana’s toolchain and VM.
However, it is still left unclear if eBPF is great for contracts target. More research is needed here.
Limiting loops is not really great since that prevents compiling a big chunk of software out-of-box. Not sure how much Rust would still compile into eBPF fine, and that requires more research.
However, forking eBPF is also not cheap. It would require you to make changes to rustc/LLVM. I suspect that lack of loops is the foundation of why e.g., memory checks could be elided. Messing with that may be tricky.
But if you are forking eBPF to change some little details, there is a temptation to fix more things. Shrink the instructions. Remove some unneeded instructions. But at this point you might as well design a thing from scratch that satisfies the whole. With more changes, less and less tooling, testing corpus, etc, could be used. That, however, writing an entire backend and all accompanying tools and libraries is a ton of work, and one of the main reasons why we picked Wasm in the first place.