Hello, Polkadot Community!
In my previous posts (1, 2), I gave you a glimpse into the exciting projects my team has been working on for the past 5 months. Now, I’m thrilled to dive deeper into our progress, current status, and future plans. So, So, buckle up, because we have been busy building! 
Our Journey So Far
One of our important goals is to develop small, composable building blocks with clearly defined boundaries. This approach has led us to create a variety of libraries. I’ll split this update into two main parts: firstly, I’ll discuss the libraries we’ve developed in connection with the Polkadot Provider API. Secondly, I’ll delve into the libraries specifically designed for the @polkadot-api/client, the go-to client for most dApp developers. 
Advancing the Polkadot Provider API
In my last post, I highlighted the need for a minimal, library-agnostic interface to foster interoperability and nurture a thriving ecosystem within Polkadot. I proposed a preliminary version of this API and introduced two pivotal libraries aimed at realizing this vision:
@polkadot-api/light-client-extension-helpers
Kudos to @matias for turning this library from concept to reality. The @substrate/connect extension, now implemented with @polkadot-api/light-client-extension-helpers, is a testament to his outstanding effort. He didn’t just bring the idea to life; he refined the initial API proposal into something far more robust and user-friendly.
This library is a strategic step towards making the @substrate/connect extension the first Polkadot Provider API-compliant extension. Why? To offer a template that anyone can fork to create their own light-client powered Polkadot Providers.
The light-client-extension-helpers form one part of this solution. They provide an API compliant with the Polkadot Provider, yet initially come with an empty list of accounts. Now, we’re focused on adding the other half: modular helpers for account management, along with a simplified API for transaction creation and message signing. We’ve made significant strides here, introducing tools like:
@polkadot-api/tx-helper
Previously previewed in my last blog post, this library is now fully operational. It’s a higher-order function that takes a JSON-RPC API connection, a signer, and an optional callback for user-data required by certain signed extensions. It outputs a function that accepts binary call-data and returns an extrinsic, effectively simplifying the complexity of constructing an extrinsic based on the chain’s signed-extensions.
While documentation is still pending, you can see this library in action in this experiment and our integration tests.
@polkadot-api/metadata-builders/view-builder
The @polkadot-api/metadata-builders offers various utilities for creating different kinds of “builders” from metadata. A key feature is the view-builder, an introspective tool essential for chain interactions. It’s particularly effective for examining calls from metadata, enabling easy creation of user interfaces that display structured transaction data for user confirmation.
Expanding the Toolkit with @polkadot-labs/hdkd and @polkadot-labs/hdkd-helpers
@matias has been instrumental in developing @polkadot-labs/hdkd and @polkadot-labs/hdkd-helpers, libraries focused on Hierarchical Deterministic Key Derivations. For a hands-on demonstration of their capabilities, check out this interactive demo.
With these additions, we’re now fully equipped to integrate account management into the @substrate/connect extension. 
But that’s not all! We’ve made more strides in relation to the Polkadot Provider API:
@polkadot-api/legacy-provider
The development of @polkadot-api/legacy-provider has been a remarkable achievement. We’ve always believed it was feasible to create a library that could detect whether the user has installed Polkadot extensions and/or the @substrate/connect extension, providing a compliant Polkadot Provider API using behind the scenes “the legacy system”. This library ensures that even in the absence of any extensions, users still get a Polkadot Provider API (albeit with an empty list of accounts). If there are injected extensions, it offers a mechanism to connect/disconnect from these, reflecting changes in the available account lists across different chains. See how it works in this demo dApp.
@polkadot-api/node-polkadot-provider
Thanks to @ryan’s efforts, we now have a provider for using the Polkadot-API client in NodeJs 
. Notably, this library internally leverages the @polkadot-api/tx-helper previously discussed 
.
@polkadot-api/json-rpc-provider-proxy
Last but not least, responding to @tomaka’s valuable feedback, we’ve released @polkadot-api/json-rpc-provider-proxy. This proxy is designed to hide the transportation layer status from the public API of the JSON-RPC provider, adding an extra layer of abstraction for developers.
Top Level Client efforts
@polkadot-api/substrate-client
The @polkadot-api/substrate-client is the bedrock of our entire framework. This low-level library offers a streamlined interface for the new JSON-RPC API specification, focusing specifically on the chainHead and transaction function groups. It adeptly handles the intricacies of nested subscriptions and error propagation without any preset behaviors or enhancers – just a barebones, yet powerful, API. You can explore its practical applications in various scenarios here, here, and here.
@polkadot-api/client/observableClient
Building on the minimalistic approach of @polkadot-api/client, the observableClient enhances its capabilities. It introduces a suite of useful behaviors like automatic unpinning and error-recovery. This more opinionated client uses an RxJS Observable-based API, facilitating the management of various cross-cutting concerns. It produces a client rich with observables such as metadata, runtime, finalized, etc., crucial for constructing the top-level client of @polkadot-api/client. Notably, this observable client is also integral to other libraries like tx-helper and legacy-polkadot-provider.
@polkadot-api/substrate-bindings
This library serves as the vital link for Substrate interactions. It provides:
- An extended version of
scale-ts, including Substrate-specific codecs. - Tools for encoding/decoding storage keys and results.
- Type definitions for descriptors used in the top-level client.
- Substrate-specific hashing functions.
@polkadot-api/metadadata-builders
A particularly fascinating library, it comprises higher-order functions that take scale-decoded metadata as input. These functions offer various utilities:
lookupHelper: Normalizes the denormalized data structure in the metadata’s lookup section.staticBuilder: Generates static TypeScript code for different chain interactions.dynamicBuilder: A dynamic version of the static builder.checksumBuilder: Produces a checksum for validating chain interactions in a given runtime.viewBuilder: Already discussed, this facilitates introspection of chain interactions.
@polkadot-api/cli
The development of @polkadot-api/cli has been a journey of adaptation and innovation, led by @ryan. Initially, I envisioned a web-based UI where developers could select specific chain interactions before generating their corresponding types. This approach aimed to streamline the development process, allowing for a more intuitive selection of necessary functionalities.
However, as we progressed, we realized the potential benefits of an interactive CLI, instead. This shift was driven by the desire to enhance the developer experience, making it more engaging and efficient. We embarked on developing an interactive CLI, anticipating it would offer a superior UX/DX (User Experience/Developer Experience).
Despite @ryan’s dedicated effort, we encountered significant challenges in delivering an optimal user experience through the CLI. The complexity of creating a user-friendly and efficient interactive CLI proved greater than anticipated. After considerable work and evaluation, we decided to pivot from our original plan.
We ultimately adopted a different approach, as detailed in this discussion. This new strategy involves a more streamlined and less interactive method, focusing on efficiency and simplicity. While we have set aside the interactive CLI concept for now, we remain open to revisiting and refining our approach in the future.
@polkadot-api/client
The flagship library for dApp developers, @polkadot-api/client is designed for creating multichain and light-client-first interactions. While still a work in progress, it’s already functional for developing basic dApps.
Looking Ahead: Charting the Future Path
As we reflect on our journey, it’s evident that our team’s passion and commitment to delivering a light-client-first alternative to PolkadotJs have been the driving forces behind our rapid and high-quality output. It’s a thrilling time for us, filled with excitement and clear objectives.
However, amidst the significant transformations at Parity and the challenges of a global team spread across time zones, we’ve reached a crucial realization: our current pace is not sustainable. Working with a small but incredibly motivated team of just three developers in different time zones has been rewarding, yet demanding. My extended working hours to align with different time zones, while enjoyable, are not a long-term solution. Recognizing this, we’re planning to divide our efforts into two distinct directions:
-
Completing the
@substrate/connectExtension Transformation@matias, with his deep involvement and expertise in the substrate-connect project, is ideally positioned to spearhead this transformation at Parity, alongside the talented @ryan. With their combined skills, the completion of this extension is not just a possibility but a near certainty.
-
Polkadot-API: Achieving Stability and Maturity
I’m excited to announce that I will be spearheading the Polkadot-API initiative externally, thanks to Parity’s support. Joining me in this venture is a highly skilled developer with whom I’ve been collaborating on OSS initiatives for years, conveniently located nearby, which will undoubtedly enhance our productivity and morale. This decision, made voluntarily and strategically, aligns with our mission’s values of fostering competition and cooperation among library authors.
Our immediate focus for Q1 2024 is on stabilizing the various libraries, adding comprehensive documentation and tests, and releasing the beta version of the top-level client. We also plan to draft an RFC proposal for the Polkadot Provider API, all while maintaining close engagement with the community through biweekly demos for feedback and ideas.
By the end of Q2 2024, our aim is to have a fully stable version of the top-level client.
Looking beyond these milestones, our ambition knows no bounds: development tools, performance enhancements, maintenance, dApp templates, migration guides from PJS to Polkadot-API, support for archive functions, a reactive framework, and much more. But, as always, our course will be steered by the needs and aspirations of our incredible ecosystem.
I look forward to sharing more details in the forthcoming OpenGov proposal and to your valuable feedback and ideas. Here’s to a future brimming with possibilities! 
A Heartfelt Acknowledgment:
As we journey through this ambitious project, it’s crucial to recognize those who have been instrumental in our progress. Their contributions, support, and insights have been invaluable.
-
@wirednkod: His unwavering support and hands-on assistance have been pivotal. From aiding in our initial zombienet integration tests to crafting examples using the
viewer-builder, his involvement and feedback have significantly shaped our libraries from the early stages. -
@peetzweg: His proactive approach in offering help, ideas, and support has been remarkable. By creating a dApp with PJS, which I could then adapt to Polkadot-API, he facilitated an enriching experience and set a precedent for future experiments. His diligence in ensuring that Polkadot-API is at least at pair with PJS’s functionalities does not go unnoticed. A massive thank you to @peetzweg!
-
@bkchr: I’m grateful for his assistance in demystifying how signed-extensions work and for being a reliable source of answers to my myriad questions.
-
James Wilson and @lexnv: Their support, feedback, and guidance, especially from the Rust perspective, have been incredibly helpful and are deeply appreciated.
-
@tomaka: His prompt responses to our issues in both smoldot and the JSON-RPC spec repository have been crucial in keeping our project moving smoothly.
-
@gavofyork: His patience in answering my questions, valuable feedback after a very early demo, and support for my decision to spin off the Polkadot-API project from Parity have been tremendously encouraging.
Each of you has played a significant role in our journey, and for that, I extend my deepest gratitude. Your contributions are a testament to the collaborative spirit of this project and the wider community.