Does it mean that the RPC state-related methods will be deprecated and eventually removed? And will it be impossible to access raw storage and storage proofs from ‘userland’ (outside a node or light client)?
Another angle (though I haven’t thought deeply about it) is to have the ability to execute the view functions in ‘userland’ and update them when there’s an upgrade, so you can rely on the current way of subscribing to the affected storage keys and keep the external API it as it is.
Direct storage access is unlikely to be ever removed as it will break existing applications.
I see that the conversation has now moved/focused on the technical details of the implementations proposed in this thread. However, I will still post my answer in the question asked from Kian (regarding pain points we have experienced) in case it is helpful for anyone.
I am focusing on implementing breaking changes in Sidecar’s endpoints and one available endpoint is staking. Quite some months ago, there were some breaking changes in the staking pallet and this was a major pain point for me.
My pain point was to understand the logic of the latest staking changes, the motivation behind those changes from a runtime/performance perspective and then determine how I would retrieve the information needed and asked from a user/tooling perspective.
If I understand correctly, here you are discussing how we will access the info needed. But who decides what information is needed or important? These view functions sound very interesting and super useful but what exactly will we be viewing? Who decides what is the valuable info for the user/external teams? From what I have seen from the staking changes, the information we get from storage calls are not necessarily what we need in the tooling side. To finally get the info needed, someone has to:
To mention a concrete example : a team wants to see if the rewards of a nominator were claimed or not. This is a simple question for a user but not as simple to retrieve with storage calls. Someone needs to know:
For me, it is important to understand the questions users will ask, their needs in data retrieval aspect, and be able to provide that information easily. From what I gather, the core developers are focused on making the chain faster and more efficient. I’m not sure how aligned these two sides are, or if sometimes the goals of one side might collide with those of the other.
Another pain point I would like to mention is communication. When a core developer explains breaking changes, it is often done from a chain-runtime perspective. However, a person reading those changes does not necessarily share the same perspective. Taking the same example with the staking changes, when a user asks if a validator claimed their rewards, they can mean :
From a core dev perspective, this question typically includes both types of rewards so the distinction is not emphasized. From a user perspective though, it can be the first case or both, and the distinction needs to be made and clearly explained. What I found helpful and a solution to this communication issue was to ask a lot more questions and avoid making assumptions—especially the assumption that the person I’m speaking with shares my perspective.
Having said all that, I need to highlight that regarding those staking changes the help/patience and multiple explanations from @ankan were huge and invaluable and I am very grateful for.
You identified the biggest weakness of view functions: They have to be defined by the runtime developers which are not necessarily the same as the people querying the data. However, this is in purpose. As you also pointed out the people querying the data often don’t have enough information to understand how to retrieve the information. This information mostly lives in the pallet developers head. This is precisely why we want to make them define the view functions.
How do they know which queries to define? You tell them what you need and why and they will add the query. The key point here is that view functions will be immutable. We are not allowed to break the interface. So this has to be done by someone with enough skill and foresight. And has to be green lit by the person responsible maintaining compatibility for a very long time.
I think this will alleviate a lot of the pain points people are having right now. However, it is a permissioned process and will need a runtime upgrade every time. For more advanced teams who need to iterate quickly we will have XCQ at a later point.
The key take away is that both things are stable. We will not break once they are stabilized.
This is a very good point.
I think we have been too lenient on pallet developers to seem themselves detached form Tooling/SDK developers, and therefore not think about this. Going forward, they should.
Any pallet developer should ensure that they expose the right set of view functions that are useful for Tooling/SDK developers. They should give their PRs enough visibility and time to collect feedback about this. Once the tooling collective is in place, this collective can have a secretary that will ensure these changes are communicated.
You as an SDK developer should also feel empowered to communicate your wishes about which SDKs are actually useful to pallet developers.
From what I gather, the core developers are focused on making the chain faster and more efficient. I’m not sure how aligned these two sides are, or if sometimes the goals of one side might collide with those of the other.
Historically yes, but admittedly we are getting more and more open-minded about this too. For example, in the very same staking pallet we now store MinActiveStake, which is s totally useless storage item for the chain, but it helps UIs. You should make requests for similar queries, and communicate it to me, @ankan or other SDK developers.
@Alex also put it in a very good way, which I will repeat as a TLDR:
How do they know which queries to define? You tell them what you need and why and they will add the query. The key point here is that view functions will be immutable.