New JSON-RPC API: Q4 2024 Update

Tech Talk
1

Hi there :wave:

Just over a year ago, @tomaka shared a detailed Q&A outlining the state of the new JSON-RPC API specification. In the 15 months since then, much has evolved and improved in that area. Now, as we enter Q4 2024, we’d like to build on that and provide an updated overview of the ecosystem’s status regarding the adoption of the JSON-RPC spec.

First, we want to clarify that the JSON-RPC spec is not a “PAPI spec” or something exclusive to our library. This specification concerns the entire Polkadot ecosystem, and the goal is to eventually deprecate the legacy RPC methods in favor of the properly specified ones.

Polkadot-API is a project that has fully embraced this challenge, and our library only works with the new JSON-RPC methods. That being said, the new JSON-RPC API is very powerful, and it is possible to build middlewares that enable all sorts of behaviors: fixing bugs when communicating with a buggy RPC implementation, handling upstream disconnections/errors without having to kill the downstream JSON-RPC connection, falling back to using the legacy RPC methods for certain actions, etc.

At PAPI, we have developed several middleware solutions so that PAPI DApps can work with some faulty and/or non-compliant JSON-RPC providers. This is necessary because many JSON-RPC endpoints are not fully compliant with the new JSON-RPC API—often due to public RPCs using outdated versions of the Polkadot-SDK node, or because even the latest release of the Polkadot-SDK node has a few problems.

It’s worth pointing out that there is no need to use these enhancers with the light client, because if we encounter an issue with smoldot, we usually have a fix before we even have time to think about how to solve it with middleware. This is one of the many advantages of using the light client: you can just upgrade a library on your DApp.

However, despite our efforts to ensure PAPI works flawlessly with endpoints that are not fully compliant, some chains are using versions of the Polkadot-SDK node that are so outdated that the PAPI middleware cannot “translate” them to the new JSON-RPC API.

Interoperability is one of the pillars of the Polkadot ecosystem, and for the ecosystem to thrive, it’s essential that its chains embrace the standards that facilitate this interoperability. The new JSON-RPC spec is a step forward towards the standardization we need to create interoperable systems. Chains that still expose non-compliant JSON-RPC APIs are holding us all back.

In this post, I’d like to provide an overview of the current state regarding the adoption of the new JSON-RPC spec:

Light Client

Polkadot-SDK Node

  • The latest release of the Polkadot-SDK has a few issues that will be addressed in the upcoming release. However, many chains are still using rather old versions of the Polkadot-SDK node, which is why the PAPI enhancers are still recommended.

RPC Providers

Many RPC providers have developed their custom gateways/middleware. I’m glad to say that, as of today, all those gateways seem to be working very well with the new JSON-RPC API. There’s a tiny issue still pending: some of these gateways assume that the id field of the JSON-RPC message is a numeric value. This is probably because PolkadotJS always used numbers for the IDs. However, the JSON-RPC spec clearly states that the id field is:

An identifier established by the Client that MUST contain a String, Number, or NULL value if included. If it is not included, it is assumed to be a notification. The value SHOULD normally not be Null and Numbers SHOULD NOT contain fractional parts.

We have created a simple middleware on our end to overcome this issue while we wait for the affected endpoints to resolve the problem on their side.

I also want to thank all the different RPC providers for how responsive and accommodating they’ve been to our requests :clap:.

Subway

The Acala team built a very impressive gateway named Subway, which reduces server load (and cost). However, this gateway was designed for the legacy JSON-RPC API. While it’s entirely possible to create a similar gateway for the new JSON-RPC API, it makes more sense to tackle that as a separate project. This is a huge opportunity for anyone eager to help the ecosystem move forward. It would be amazing to see a similar gateway for the new JSON-RPC API.

Parachain Teams

The public JSON-RPC endpoints of some parachain teams expose an archaic version of the JSON-RPC API that our middleware can’t bridge.

We have created the following repository to keep track of the problematic endpoints: polkadot-api/compliant-RPCs.

In the case of Polkadot, we have also opened issues on all the GitHub repositories of the problematic teams.

We have identified 13 problematic Polkadot parachains whose JSON-RPC endpoints don’t expose a modern version of the JSON-RPC API. Among these, there are only three for which we can use the light client to access the modern JSON-RPC API. For the rest, either they didn’t expose up-to-date chain specs, or they did, but their full nodes were buggy.

As of today, the problematic Polkadot parachains are:

  • Aventus: We couldn’t find their chain specs, so we haven’t been able to test if this parachain works with the light client. GitHub issue.
  • Composable: The chain spec they provide does not have up-to-date bootnodes, so we don’t know if this parachain works with the light client. GitHub issue.
  • Continuum: Again, no chain specs found, so we don’t know if it works with the light client. GitHub issue.
  • Energy Web: We couldn’t find the GitHub repository for their chain :man_shrugging:.
  • Interlay: This parachain works with the light client. GitHub issue.
  • InvArch: It also works with the light client, and the team is addressing the issue. GitHub issue.
  • Parallel: There seems to be an issue with their full nodes that causes the light client not to work properly on this parachain. We have opened two issues (1, 2) in their repository. If I had to make a wild guess, I’d say this parachain is completely abandoned.
  • Pendulum: It works with the light client. GitHub issue.
  • Polkadex: Outdated chain specs, so we don’t know if it works with the light client. GitHub issue.
  • Robonomics: It works with the light client. GitHub issue.
  • Sora: We couldn’t find their GitHub repository, so we don’t know if it works with the light client and couldn’t report an issue.
  • Subsocial: Their full nodes seem to have the same problem as Parallel’s, so it doesn’t work with the light client. GitHub issue.
  • t3rn: Outdated chain specs, so we don’t know if it works with the light client. GitHub issue.

We have created a GitHub repository polkadot-api/compliant-RPCs to track the compliance of all known public RPCs with the JSON-RPC specification for Polkadot, Kusama, Paseo, and their parachains. It’s worth noting that there are many other problematic parachains on Kusama and even a few on Paseo.

Call to Action: Let’s Push for Compliance

If you are close to any of these teams, please kindly encourage them to improve their chains so that they become compliant with the new JSON-RPC API. Doing so will bring numerous benefits for their chain and for those trying to build on top of it. Some of these benefits include:

  • Enabling DApp developers to easily interact with their chain using PAPI.
  • Unlocking improved performance and reduced resource usage.
  • Providing out-of-the-box strong type generation for your chain.
  • Allowing developers to handle runtime upgrades on the fly.
  • Soon, having their chains work flawlessly with the PAPI dev-tools that we are currently working on.
7 Likes