TL; DR:
Discover the Validator Selector Tool, the first light-client only DApp built with Polkadot-API, based on the award-winning research lead by @jonas. If you’re involved in nominating validators on Polkadot, this tool is a game-changer. Read on to uncover why this achievement is more significant than it may initially appear.
The Inception
About 9 months ago, @jonas from W3F reached out on a Matrix channel seeking assistance to develop a DApp. The goal was to transform the findings of his team’s research into a practical application. Intrigued by the abstract, I eagerly volunteered for the project, with one condition: the DApp must exclusively utilize the light-client. Jonas, excited yet cautious, informed me of the significant challenge ahead, particularly the immense data required from the blockchain. Nevertheless, we agreed that if we could efficiently process this data in the background, allowing users to focus on selecting their preferences regarding their validator choices in the meanwhile, it would be feasible.
Our journey commenced, and we were fortunate to have Daniel Kalman on board, who played a pivotal role in enhancing the user experience (UX) design. Alongside him, we had the invaluable support of Grzegorz Miebs from the research team. Grzegorz was instrumental in handling the more intricate tasks – he meticulously developed and structured the essential data models and files, a critical process that enabled us to seamlessly integrate these components into the DApp.
Facing the daunting task of data collection, I explored numerous strategies. Even utilizing a centralized RPC server was challenging, but attempting this with the light-client was an entirely different ordeal due to several issues:
- Initially, using Polkadot JS (PJS) with centralized RPC nodes was somewhat manageable, but replicating this with the light-client provider (the
ScProvider) proved to be a dead end. Despite attempts at custom solutions, success eluded me. - Curious if the problem stemmed from PJS’s lack of support for the new JSON-RPC API, I turned to CAPI. Dishearteningly, not only was CAPI incompatible with the new JSON-RPC API, but it also failed to facilitate timely data collection, even with centralized RPCs. Despite my efforts and consultations with core developers, a viable solution using CAPI remained out of reach.
- Digging deeper into the light-client’s limitations, I identified a bug in smoldot, which hindered downloading essential
Staking.Nominatorsentries needed for calculating validator votes. - Additionally, smoldot’s recent API update, which allowed running on a WebWorker, hadn’t been integrated into the
@substrate/connectpackage yet, missing out on potential improvements. - Compounding these challenges, the
@substrate/connectextension was underperforming due to the fact that smoldot was no longer running from its background/service-worker script.
This context sets the stage for the obstacles we faced about eight months ago, as we strove to deliver the “Validator Selector” DApp as a light-client MVP.
Initial MVP
Despite all these challenges, my resolve to create a truly decentralized application using the light-client with smoldot remained unshaken. To demonstrate this possibility, I crafted an ad-hoc solution using an experimental library I was exploring at the time, unstoppablejs. This approach, albeit more of a workaround, was instrumental in proving that with appropriate tooling, a fully functional DApp based on the light-client was achievable.
Our efforts culminated in a nice MVP of the DApp, which was presented by @jonas at Polkadot Decoded 2023.
This achievement was significant, yet I was acutely aware that we were still far from providing the average DApp developer with the necessary tools to craft a truly decentralized DApp. The key limitations included:
- Due to a bug in smoldot, the DApp had to “cheat” to obtain the list of nominators, relying on a centralized RPC server.
- For performance optimization, I bypassed the
@substrate/connectpackage, directly integrating smoldot with the WebWorker in my custom solution. - The DApp did not utilize dynamic codecs from the runtime, which limited its ability to ascertain compatibility with the current runtime for retrieving storage items.
- The application was not yet leveraging the new JSON-RPC API.
- A desirable feature, allowing users to create an extrinsic to update their nominators based on the DApp’s recommendations, was not feasible yet.
- Generally, the entire process of collecting validator data was a makeshift solution, intended more as a proof of concept than a final product.
This stage of development laid the groundwork for what was needed to move towards a more refined and truly decentralized DApp.
Today
The landscape has evolved significantly since Decoded 2023. Shortly after the event, Parity offered me the chance to lead a small, dedicated team including @matias and @ryan. Our mission was ambitious: to develop a light-client first alternative to PJS. This opportunity, while exciting, meant I had to pause my involvement with the “Validator Selector” tool.
I’ve previously shared updates about the progress my team has made with Polkadot-API. Yet, I was eager to demonstrate something tangible, something that showcases the strides we’ve made over these past months…
Recently, I finally carved out time to migrate the original, somewhat “forgotten” MVP of the “Validator Selector” tool to Polkadot-API. The results of this migration, completed just a couple of days ago, are nothing short of exhilarating.
Key highlights of the migration include:
- Simplified Codebase: The transition from the ad-hoc solution to Polkadot-API has made the codebase more maintainable and understandable, as evidenced by the commit which adapts the source code (166 additions vs. 547 deletions
). - Truly Decentralized: The DApp now entirely relies on the light-client for collecting all the data, eliminating previous workarounds.
- New JSON-RPC API Integration: All light-client interactions are now conducted via the new JSON-RPC API.
- Dynamic Codecs and Runtime Compatibility: The DApp adapts to runtime changes and verifies compatibility with the current runtime.
- Preparation for Future Upgrades: It’s possible to prepare the DApp for a future Runtime upgrade, like we did in here, preparing the DApp in case that an update of the Identity pallet (currently available in Rococo) makes it to the Polkadot relay chain. This also makes the DApp to be curretnly compatible with both runtimes.
- Enhanced
@substrate/connectUtilization: The improvements made on@substrate/connectnow enable the integration with the new WebWorker API, ensuring a much better performance than in the past, even when the@substrate/connectextension is not present. - Exceptional Development Experience and Performance: The type system and development experience are incredibly advanced. Bundle sizes are super tight, and Lighthouse performance scores are unprecedented, a notable achievement given the additional downloads required for users without the
@substrate/connectextension (namely: the smoldot binaries and chainspecs). - Ease of Feature Integration: Adding a new functionality for users to update their validator nominations within the same DApp is now straightforward, a feature I plan to implement soon.
Lighthouse reports showcase the remarkable improvements:
Desktop Report (without the @substrate/connect extension):
Mobile Report (without the @substrate/connect extension):
This migration to Polkadot-API didn’t just enhance the DApp; it also aided in identifying a bug in the JSON-RPC server implementation of the Polkadot-SDK node. This discovery was made while comparing data loading speeds against centralized RPC servers, highlighting the robustness and reliability of our new approach.
Not Everything is Perfect (Yet)
There are a few aspects that still require refinement:
-
Data Loading Time without the extension: Currently, when the
@substrate/connectextension is not installed, loading all the necessary data to present results to the user can exceed a minute. In reality, if users are thoroughly engaging with the survey questions, this time frame might align with their completion time. However, there’s certainly potential for optimizing this process to be more efficient. -
Handling Large Data Deliveries: An unexpected challenge we’ve encountered is smoldot delivering the data for approximately 50,000 nominators in a single message. This approach differs from the anticipated progressive data delivery, leading to temporary freezing of the DApp as it synchronously decodes all these nominators. To address this, we need to implement more graceful data handling within PAPI. Additionally, we’ll be reaching out to smoldot to discuss the feasibility of progressively delivering large data sets to prevent such issues.
-
Updating the List of Validators: The feature to submit an extrinsic for updating the list of validators is still in the works. Although I’ve developed a prototype on a local branch, its UX is currently subpar. I am committed to refining this feature and will update the DApp with a more user-friendly version of this functionality as soon as it meets our standards for a decent UX.
The Imminent Future
The journey thus far has been nothing short of extraordinary. I am convinced that we are crafting something remarkable with these developments. Nevertheless, I recognize the critical need for stabilizing these APIs and providing comprehensive documentation. It’s essential for our vibrant community to have the opportunity to experiment with these tools as soon as possible. Therefore, our primary objective for the end of Q1 is to release the first Beta version of the @polkadot-api/client. This release will include basic documentation, enabling others in the ecosystem to begin exploring and utilizing it.
Please Help Us Continue This Work
As I mentioned in a previous update, my commitment to accelerating the development of Polkadot-API is unwavering. This dedication has led me to a significant decision: I will be leaving Parity at the end of this month. My focus will be solely on advancing Polkadot-API, alongside another exceptional developer, Victor Oliva. We’re currently preparing an OpenGov proposal, which we plan to submit shortly. I will share the proposal here as soon as it’s ready. Your support is crucial – I kindly ask for your “aye” on the proposal to help us sustain and advance this vital work.
I mean, PAPI is still under development and we are releasing experimental and non-stable versions in order to easily test our work.
.