There are several pressing problems including the redesign of the events system, having fast non-persisted values during block import, managing complexity in the transactional storage system and dealing with large code blobs in PoVs, as well as paving the way to useful GPU utilisation. These problems are all related to our over-reliance on a single large fixed persistent cryptographic data structure - the State Trie.
New storage primitives are needed in order to solve each of these issues effectively and efficiently. A new issue has been opened in Github putting forward a proposal for a minimal new host interface which should allow for this to happen. This may be of interest to the Polkadot builder community.
We should’ve fast non-persistent data for roughly arbitrary rust data types.
We’d have a rust Allocator backed by wasm allocating memory pages, with the first containing any static muts from the wasm.
These pages would be regular allocations during import which simply survive through each wasm entry point, and make the wasm pointers deterministic. We’d build the block by making these pages be copy-on-write shared memory in each new entry point, so any changes can be rolled back if a transaction fails.
We’ve a concern that wasm pointers placed into these pages could point into other memory which we do not persist across wasm entry points, like stack or another Allocator. We could fork alloc::{vec,collections} to constrain types to not contain such pointers, but doing this maybe excessive, given this simply causes deterministic segfaults.