Everyone!
Following my initial input in the Developer Experience must be our #1 Priority discussion, I’m here to elaborate on a significant advancement that I believe is crucial for the enhancement and expansion of our ecosystem: the Polkadot Provider API.
Why Do We Need the Polkadot Provider API?
Our ecosystem has a gap. We’re missing a standardized, simple convention to address some cross-cutting concerns for dApps, such as:
- Finding available chains for dApp connection.
- Proposing new chains for dApp connectivity.
- Seamlessly connecting to a chosen chain via a minimal interface with the new JSON-RPC API.
- Identifying which accounts are accessible for the connected chain(s).
- Simplifying transaction creation by only requiring the transaction method and arguments, with the Provider taking care of the complexities (signed-extensions with their respective data, signing the transaction, checking if the transaction would raise errors, etc, etc).
The Impact and Opportunities of This API:
The introduction of this minimal and library-agnostic convention will pave the way for numerous interoperable opportunities:
- Creation of PolkadotJs Alternatives: Empowering library authors to develop alternatives to PolkadotJs.
- Wallet Flexibility: Enabling wallets to implement the Polkadot Provider API, freeing them from tight coupling to a PolkadotJs specific API.
- Enhancer Development: Facilitating the creation of enhancers to handle transaction complexities, simplifying integration with Ledger, WalletConnect, etc.
- Native App Support: Allowing the development of native Android/iOS apps with webviews to load compliant dApps.
- Progressive Web App Providers: Enabling the creation of Polkadot Providers for Progressive Web Apps.
- Seamless Wallet Integration: Allowing the smooth integration of favorite dApps into wallet interfaces.
- And much more!
Current Limitations and the opportunity ahead of us:
Our ecosystem is somewhat limited by PolkadotJs (PJS), because it necessitates all wallets to adapt to a less-than-ideal API for creating transactions. This limitation hinders the seamless communication and interoperability, creating a roadblock for expansive and inclusive dApp and wallet development.
That being said, I have immense gratitude and respect towards PJS and its creator and maintainer, Jaco. Although, its current design limitations may seem evident in hindsight, it could not have been foreseen when PJS was initially developed.
Nevertheless, we’ve reached a pivotal point in the creation of building truly decentralized applications. Mass light client adoption can only be achieved by building the proper tooling that can be integrated with new light-client friendly JSON-RPC spec; something that PolkadotJs is unable to do. As a result, it’s paramount that we center our efforts around a streamlined, unified abstraction that address the earlier mentioned cross-cutting concerns, hence the introduction of the Polkadot Provider API.
The Polkadot Provider API will stand as a beacon of growth, inclusion, and seamless operations for our ecosystem, providing a future of enhanced interoperability, flexibility, and overall development.
Polkadot Provider API proposal:
For clarity, here are the TypeScript definitions (with some comments) encapsulating the envisioned API:
type Callback<T> = (value: T) => void
type UnsubscribeFn = () => void
interface PolkadotProvider {
// Retrieves the current list of available Chains
// that the dApp can connect to
getChains: () => Promise<Chains>
// Registers a callback invoked when the list
// of available chains changes
onChainsChange: (chains: Callback<Chains>) => UnsubscribeFn
// Allows the dApp to request the Provider to register a Chain
addChain: (chainspec: string) => Promise<Chain>
}
// The key is the genesis hash of the chain
type Chains = Record<string, Chain>
interface Chain {
genesisHash: string
name: string
connect: (
// the listener callback that the JsonRpcProvider
// will be sending messages to
onMessage: Callback<string>,
// the listener that will be notified when the connectivity changes
onStatusChange: Callback<ProviderStatus>
) => Promise<ChainProvider>
}
type ProviderStatus = "connected" | "disconnected"
interface ChainProvider {
// it pulls the current list of available accounts for this Chain
getAccounts: () => Accounts
// registers a callback that will be invoked whenever the list
// of available accounts for this chain has changed
onAccountsChange: (accounts: Callback<Accounts>) => UnsubscribeFn
// contains a JSON RPC Provider that it's compliant with new
// JSON-RPC API spec:
// https://paritytech.github.io/json-rpc-interface-spec/api.html
provider: JsonRpcProvider
}
type Accounts = Array<Account>
interface Account {
// SS58Formated public key
publicKey: string
// The provider may have captured a display name
displayName?: string
// `callData` is the scale encoded call-data
// (module index, call index and args)
createTx: (callData: Uint8Array) => Promise<Uint8Array>
}
interface JsonRpcProvider {
// it sends messages to the JSON RPC Server
send: Callback<string>
// it disconnects from the JSON RPC Server and it de-registers
// the `onMessage` and `onStatusChange` callbacks that
// were previously registered
disconnect: UnsubscribeFn
}
These definitions articulate the framework and essential structure we need for this proposed direction.
Addressing Potential Questions
You might logically wonder about the practicality of creating a new library based on an interface with no current implementations. Also, will the new libraries built around this Polkadot Provider not communicate with existing PolkadotJs wallets?
Incorporating Existing Tools
Something exceptional about the Polkadot Provider API is its minimalism, allowing the integration of existing tools seamlessly. For example, a library could be (and will be) developed to expose a function taking a PolkadotJs InjectedExtension as an argument. The library would then perform the necessary wiring using substrate-connect behind the scenes to produce a compliant Polkadot Provider API.
Even without extensions, a Polkadot Provider can still be established internally using substrate-connect, allowing dApps to establish necessary connections, despite always returning an empty list of Accounts for all Chains.
Expanding to Various Environments
The versatility of the Polkadot Provider API extends to creating providers for node.js or any other environment, demonstrating its wide-reaching applicability.
Our Efforts at Parity
What are we doing at Parity to crystallize this vision?
We are in the process of developing a composable, modular, and “light-client first” alternative to PJS, fundamentally built on the Polkadot Provider API.
In addition, my team is dedicated to empowering the community by offering the essential building blocks for creating these kinds of Providers. We are actively developing two different libraries, aiming for a swift release to expedite the enhancement of the Polkadot ecosystem.
I would like to share with the community the APIs of these 2 different libraries:
@polkadot-api/light-client-extension-helpers
This library is meant to be used from within extensions and it basically encapsulates all the challenging problems that have been solved over the years from the @substrate/connect extension. The library will expose four primary components:
- backgroundHelper: which will be a function that must be imported and registered into the background of the extension.
- contentScriptHelper: a function that will handle the communication between the tab’s webpage and the extension’s background script.
- webPageHelper: an interface meant to be used from the tab’s webpage.
- extensionPagesHelper: a group of functions that are meant to be used from within the extension’s page.
Let’s dig deeper into the responsibilities and the APIs of these different components:
backgroundHelper
The core logic that will be running in the background. Internally, it will register an instance of smoldot, keep the relevant-chains persisted into storage, keep the sync snapshots of those chains up to date, manage the active connections and bootnodes, etc. All this while communicating with the content-script and the rest of the APIs from the extension.
It’s API is fairly straight-forward:
export type BackgroundHelper = (
// A callback invoked when a dApp developer tries to add a new Chain.
// The returned promise either rejects if the user denies or resolves if the user agrees.
onAddChainByUser: (input: InputChain, tabId: number) => Promise<void>
) => void
export interface InputChain {
genesisHash: string,
name: string,
chainspec: string
}
contentScriptHelper
It will be a void function that will handle the communication between the tab’s webpage and the extension’s background script.
webPageHelper
When imported from within the web-page it will expose the following interface:
import type { ProviderStatus } from "@polkadot-api/json-rpc-provider"
type Callback<T> = (value: T) => void
type UnsubscribeFn = () => void
export interface LightClientProvider {
// Allows dApp developers to request the provider to register their chain
addChain: (chainspec: string) => Promise<RawChain>
// Retrieves the current list of available Chains
getChains: () => Promise<Record<string, RawChain>>
// Registers a callback invoked when the list of available chains changes
onChainsChange: (chains: Callback<RawChains>) => UnsubscribeFn
}
// The key is the genesis hash
type RawChains = Record<string, RawChain>
export interface RawChain {
genesisHash: string
name: string
connect: (
// the listener callback that the JsonRpcProvider will be sending messages to.
onMessage: Callback<string>,
// the listener that will be notified when the connection is lost/restablised
onStatusChange: Callback<ProviderStatus>
) => Promise<JsonRpcProvider>
}
export interface JsonRpcProvider {
// it sends messages to the JSON RPC Server
send: Callback<string>
// it disconnects from the JSON RPC Server and it de-registers
// the `onMessage` and `onStatusChange` callbacks that was previously registered
disconnect: UnsubscribeFn
}
extensionPagesHelper
An interface with a set of functions to allow the extension page to manage the persisted chains, their connections and boot-nodes. Its API is as follows:
import type { GetProvider } from "@polkadot-api/json-rpc-provider"
export interface LightClientPagesHelper {
deleteChain: (genesisHash: string) => Promise<void>
persistChain: (chainspec: string) => Promise<void>
getChain: (genesisHash: string) => Promise<PagesChain>
getActiveConnections: () => Promise<
Array<{tabId: number, genesisHash: string}>
>
disconnect: (tabId: number, genesisHash: string) => Promise<void>
setBootNodes: (genesisHash: string, bootNodes: Array<string>) => Promise<void>
}
export interface PagesChain {
genesisHash: string,
name: string
ss58format: number
nPeers: number
bootNodes: Array<string>
provider: GetProvider
}
@polkadot-api/tx-helper
A library, enabling consumers to integrate their custom UIs. This approach facilitates the safe signing of transaction data by users, capturing all pertinent information from signed-extensions.
Notably, whereas @polkaddot-api/light-client-extension-helpers is designed exclusively for browser extensions, @polkadot-api/tx-helper exhibits a broader utility, apt for different contexts such as CLI-based programs.
It’s public API:
import type { GetProvider } from "@polkadot-api/json-rpc-provider"
export type GetTxCreator = (
// The `TransactionCreator` communicates with the Chain to obtain metadata, latest block, nonce, etc.
chainProvider: GetProvider,
// This callback is invoked in order to capture the necessary user input
// for creating the transaction.
onCreateTx: <UserSignedExtensionsName extends Array<UserSignedExtensionName>>(
// The sender of the transaction
from: MultiAddress,
// The scale encoded call-data (module index, call index and args)
callData: Uint8Array,
// The list of signed extensions that require from user input.
// The user interface should know how to adapt to the possibility that
// different chains may require a different set of these.
userSingedExtensionsName: UserSignedExtensionsName,
// The user interface may need the metadata for many different reasons:
// knowing how to decode and present the arguments of the call to the user,
// validate the user iput, later decode the singer data, etc, etc.
metadata: Uint8Array,
// An Array containing a list of the signed extensions which are unkown
// to the library and that require for a value on the "extra" field
// and/or additionally signed data. This will give the consumer the opportunity
// to provide that data via the `overrides` field of the callback.
unknownSignedExtensions: Array<string>,
// The function to call once the user has decided to cancel or proceed with the tx.
// Passing `null` indicates that the user has decided not to sing the tx.
callback: Callback<null | {
// A tuple with the user-data corresponding to the provided `userSignedExtensionsName`
userSignedExtensionsData: UserSignedExtensionsData<UserSignedExtensionsName>
// in case that the consumer wants to add some signed-extension overrides
overrides: Record<string, {value: Uint8Array, additionalSigned: Uint8Array}>,
// The type of the signature method
signingType: SigningType
// The function that will be called in order to finally sign the signature payload.
// It is the responsibility of the consumer of the library (who is providing this callback),
// to decode the provided input and ensure that the data that it's been requested to sign
// matches with the previously provided `callData` and with the user provided
// `userSignedExtensions`. If there are any inconsistencies, then the returned Promise
// should reject. Otherwise, it should resolve with the signed data.
signer: (input: Uint8Array) => Promise<Uint8Array>
}>,
) => void,
) => {
createTx: CreateTx,
// it disconnects the Provider and kills this instance
destroy: () => void
}
export type CreateTx = (
from: MultiAddress,
callData: Uint8Array,
) => Promise<Uint8Array>
export type SigningType = "Ed25519" | "Sr25519" | "Ecdsa"
export type AddresType = "Index" | "Raw" | "Address32" | "Addres20"
export interface MultiAddress {
addressType: AddresType
publicKey: Uint8Array
}
type SignedExtension<Name extends string, InputType> = {
name: Name
data: InputType
}
// This list is still a WIP
export type UserSignedExtensions =
| SignedExtension<"mortality", boolean>
| SignedExtension<"tip", bigint>
| SignedExtension<"assetTip", { tip: bigint, assetId?: bigint | undefined }>
export type UserSignedExtensionName = UserSignedExtensions["name"]
export type UserSignedExtensionsData<T extends Array<UserSignedExtensionName>> =
{
[K in keyof T]: (UserSignedExtensions & { name: T[K] })["data"]
}
In Closing…
Thank you for taking the time to journey through this comprehensive post. Your attention and dedication to understanding the nuances and developments of the Polkadot Provider API are sincerely appreciated. As we work towards refining and improving these proposals, your feedback is invaluable. Please share your thoughts, suggestions, and any questions you might have. Together, we can shape the future of a brighter ecosystem.
. Nevertheless, we’ve tested the interface and identified some improvements. Below is the refined interface proposal:
Thanks for pointing this out! Yes, I agree, we should also get rid of the