Wasm view functions

Tech Talk
1

Background

In Solidity, contract developers can create view functions for their contract to help clients to access contract state.

In Substrate, we have runtime APIs that achieve similar goal. That’s totally useable and a good enough solution. However, it still have a lot of room for improvement.

One of the nice property of the Solidity view function is that the same getter can be used for both contract logic and front-end and that reduces a lot of duplicated codes. For example, a rebase token exposes a balanceOf function that returns the rebased token amount that can be changed for every block. Both other contracts and frontend/wallets are using the same balanceOf API to query balances and does not need to care where is this number stored nor how it is calculated. It just works.

We lost this nice property in the current storage based API. Most of the Substrate UI are accessing the storage directly and sometimes have to reimplement the additional calculation logic to convert them to user friendly representation. This means duplicated code between the Rust runtime logic and UI JS/TS logic. The more code, the more work, and the more bug.

Runtime API partially address by allowing runtime developers to expose Wasm runtime API interface to call some Wasm code to get the right number. This avoids the duplicated code issue. However, it also lost a nice property of using storage based accessor: change subscription.

Polkadot.js and many other UI subscribes the storage changes and able to update the changes to user in real time. This allow users always see up-to-dated value, bots able to subscript for changes in realtime. This allow developers to build efficient and elegant applications easily.

By switching to Runtime API based accessor, pulling are required to detect change. For some applications that need to monitor many/all accounts, it will completely destroy the performance.

Solution

I am here (re)proposing an idea that was floating between discussion threads for many years and finally now feasible: Wasm view functions.

The goal is simple, take the best of both worlds.

We can avoid code duplication by reusing the same Rust code implemented in the runtime and compile them to wasm function and make then available to clients.

We can detect the accessed storage when executing the wasm function and subscribe those storages for change notifications. When any of those storages changed, rerun the wasm function and resubscribe to the storages.

In this way, we can for example implements a rebase token, and notify user every time the balances changed, no matter if it is triggered by block number of a transfer or some other storage changes. We have all the dependent storage for this property and can only subscribe the required storages, no more, no less.

Implementation

I had this idea for many years but it wasn’t really feasible back then. This have changed.

With Chopsticks and smoldot, it is proven possible to:

  • Run wasm runtime in any modern JS environment
  • Create a custom backend for the storages and connect it to the wasm host

To make this working e2e, we need:

  • Rust framework to create Wasm accessor functions
  • Custom wasm execution env similar to the one in Chopsticks and collect all the accessed storages within a wasm function
  • Some helper JS library to put everything together
13 Likes
2

I see a lot of utility to this idea, and makes sense to have such options available to our ecosystem.

I just wanted to note in the context of creating community standards for Wallets or other UIs to interface with, I still believe something at the level of XCM makes more sense to me.

Feels to me that Wasm view functions would solve a specific set of problems, allowing front-ends to execute specific logic implemented inside the runtime, and I expect this will be most useful for specific app scenarios unique from chain to chain.

3 Likes
3

This is totally the case in nomination pools point/balance logic as well. Points is almost always the same as balance, and all wallets have to re-implement it.

In the short term, we will fix it with runtime API. Long term, this could be a good excuse to try out the wasm view functions as well.

I think this only applies to the Balance example and XCM is not so relevant for the broader use cases

Here’s an alternative: tap into the existing machinery for RPC-based storage update notifications. The runtime would be allowed to declare a set of “virtual storage items”. these are never stored anywhere and are instead “getter functions” as you said. They have a key in the trie as well, ergo a client can subscribe to them in the same way that it can subscribe to a normal storage item.

3 Likes
4

The runtime would be allowed to declare a set of “virtual storage items”. these are never stored anywhere and are instead “getter functions” as you said. They have a key in the trie as well, ergo a client can subscribe to them in the same way that it can subscribe to a normal storage item.

Could you expand a bit more on how this would work? If these are virtual storage items, how do they have a key in the trie?

5

Wasm view functions would be really useful for graceful upgrade of clients when some storage migration happens that client depends on by just releasing a new set of wasm view functions.

Wondering if there has been any progress on this?

6

Could you expand a bit more on how this would work? If these are virtual storage items, how do they have a key in the trie?

Truthfully I don’t exactly remember nor can I re-understand it, which means it was probably not a good idea :sweat_smile:. The idea you had and we discussed elsewhere seems better to me:

make the runtime API, record storage that was touched, then subscribe all those storage keys, upon subscription update, re-execute the runtime-api

Wondering if there has been any progress on this?

PSA to builders: A good way to express interest is to upvote [FRAME Core] Add support for `view_functions` · Issue #216 · paritytech/polkadot-sdk · GitHub with a :+1:

7

I had a working PoC but it is unusable due to restore disable-runtime-api · Issue #1621 · paritytech/polkadot-sdk · GitHub that the wasm will include the whole runtime, which makes it super large.

1 Like
8

Created a new issue based on our discussion
Subscribe-able runtime apis · Issue #3594 · paritytech/polkadot-sdk · GitHub. I think this should be achievable and very useful. Any feedback is welcome. :slight_smile:

2 Likes
9

I stumbled upon this idea once more.

I would like to try and solve this, but from a very product oriented perspective: Let’s identify a few known pain points, and build tech that exactly solves that, rather than the opposite.

Raising this, as there are multiple options:

  1. raw runtime apis, for exmaple the one proposed in Kusama: Make the current inflation formula adjustable. by kianenigma · Pull Request #364 · polkadot-fellows/runtimes · GitHub
  2. view fn, drafted in Implement pallet view function queries by ascjones · Pull Request #4722 · paritytech/polkadot-sdk · GitHub
  3. XCQ, being worked on by @xlc in GitHub - open-web3-stack/XCQ: Cross-Consensus Query Language for Polkadot

From a technical perspective, all 3 have non-overlapping properties, so a native solution is to build all 3. But let’s first ask which one is the low hanging fruit.

I have a few examples of such low-hanging fruits, but I won’t reveal them to keep the conversation unbiased.

To that end, I am writing this to make a CTA to all TS of those who have coded TS in our space to comment:

When was the time when you were working with api.query, and you felt the most pain? This pain could namely be due to:

  • You needing to re-implement (possibly changing) onchain logic in TS
  • The storage you were reading was too complicated/low-level
10

I’m spending hours these days calculating the locks for governance:

  • you’ve got locks on some tracks (bc of votes or delegation)
  • check the past casted votes
  • check the past ref, their outcome
  • compare your votes with the outcome of the ref
  • calculate the lock based on conviction
    :exploding_head:
2 Likes
11

Thank you Kian for bringing this up.

The initial most pain I received back in the day, starting with PJS API, was that there wasn’t really decent documentation of “what I pick from where”. Then once I found it, I had to combine multiple queries to receive all needed information and organise it for the purpose of the dApp.

Documentation is already resolved with strong typings, introduced by PAPI team - meaning its now easy to find what you are looking for.

The “gathering” is something that was introduced by the same team (and it makes sense) under the name “actions”. What I receive from it is that there should be a layer for “common reusable” actions, that one can run to reproduce a result in a different app (e.g. a transfer action).

I think this approach is pretty decent and should be either implemented by the TS side layer.

What I, as a TS dev would like to see - is a gathering of basic parameters on the chain level that will be easily accessible - even though maybe that suppresses the functionality that in-the-first-place pallets start to solve and allow the API layer to handle it

1 Like
12

What if we can unify those three approaches? What if it is all XCQ. Hear me out.

What if 1) and 2) just return XCQ programs instead of the value itself. I say this because 1) and 2) are mainly about discovering common queries which are already queryable via XCQ.

This might potentially leads to less code duplication and avoids the situation where stuff is only accessible via 1) and 2) but not via XCQ because somebody didn’t write an extension for it. It would allow us to confidently say that everything can be queried via a custom XCQ but also defining a list of useful queries inside the runtime.

The problem is of course that it might be complicated to define XCQ programs within the runtime. They have to be compiled for a different target. But I think if we really want that we could make it happen.

1 Like
13

The XCQ program doesn’t necessary need to be compiled into the wasm runtime. They can just uploaded to the onchain state like smart contracts. Some additional care are needed to ensure compatibility but all very manageable (e.g. make sure the programs are versioned with runtime version and before runtime upgrade someone uploads the XCQ programs for the new runtime version).
Or just some extra step in wasm builder to have them included into the wasm runtime.

So yeah it is more about if we want to do it or not.

14

Dealing with XCQ is still more complicated than just defining a view function the classical way inside the runtime. Which is the easiest way of distributing interesting queries with the runtime. Something we want in my opinion. Off chain tools accessing storage is a big no-no as they will break when storage changes.

The question is whether those view functions should be based on XCQ internally or are written the classical way.

I suggest the following plan:

  1. Add view function’s written in the classical way as an immediate remedy for the current situation. Both pallet scoped and runtime scoped ones.
  2. Develop XCQ in parallel since we still need it to express advanced use cases where we have no view function for.
  3. Discourage and deprecate storage access by Dapps as it is subject to breaking changes.
  4. Maybe change view functions to execute XCQ internally without breaking the existing interface. Only if we see a benefit in this.
3 Likes
15

Should we assume that access to storage proofs will also be deprecated? What will be the alternative?

16

Are you referring to light clients? They will access view functions in the same way as a full node: By calling into the runtime. The runtime will then generate storage accesses while executing the view function. On a light client those will continue to be fulfilled by storage proofs received from full nodes in the network.

View functions are not about removing storage accesses per se. It is a way for the runtime to abstract them away from the user.

One open question remains: How to subscribe to changes. So far this is mostly based on subscribing to changes to specific storage items. So we might have to provide a way to subscribe to changes of a specific view functions. When block times get lower we can’t afford to poll every block.

3 Likes
17

Sounds like a good plan, and the best tradeoff.

The only question we need to think about is a good syntax to support both pallet and runtime level view functions.

I think the existing view function PR is a great starting point, and all we need is a way to expand the runtime level aggregated enum to be extensible. As a side note, I always wished the RuntimeCall to also be extensible in the runtime level (imagine you want to add a pallet-less, temporary tx to the runtime), this is the same deal.

@josep / @IkerParity I hope this also resonates with you, and as you embark on building pallet-scoped APIs, you demand polkadot-sdk to give you stable view functions and use those, instead of reading storage directly. This can be a very symbiotic deal between us, let’s make it work!

Also, what do you think about the subscription issue?

18

I was referring to the RPC method state_getReadProof.

To me make sense to keep remote access to the storage and the proofs for offchain clients.

And yes, the subscriptions look challenging in the context of the view functions.

19

For subscription of runtime API (which view function will be based of) Subscribe-able runtime apis · Issue #3594 · paritytech/polkadot-sdk · GitHub

When the runtime API are executed locally, it is easy to implement subscription. Just subscribe to all the accessed storage during the runtime API execution.

20

I think it would be nice if the runtime level view functions have to be composed out of pallet level view functions instead of direct storage accesses. The reason is that otherwise those runtime level functions would need to know storage details and track when a migration takes place. We want to encapsulate that knowledge into a pallet. We could then also generate the subscribe function for them automatically as the pallet view functions already declare which storage items they access.

Agreed. We should build subscriptions into view functions from the beginning.