`subrpc`: manage multiple endpoint registries and quickly find endpoints

chevdor · January 20, 2023, 11:29am

`subrpc`

I am happy to share a new cli tool: subrpc.

Context

I recently posted about the Polkadot Network Directory which helps find chains and informations related chains and parachains.

Here is the repo:

Many projects already contributed with PRs bringing their data to the directory.

One of the benefits of having this data is that we could also generate a registry: a list of available endpoints:
https://paritytech.github.io/polkadot_network_directory/registry.json

There is however no reason for this data to be centralized in one location and users may be interested in either subsets or augmented data set.

I am happy to share the first versions of subrpc. You can check its readme for the full details.

TLDR

The TLDR is:

subrpc is a:
- cli written in Rust you can use in your terminal to find endpoints: subrpc ep get dot
- a lib/crate you can use from your app to connect to the right endpoint without having to manage that part in depth
- local database YOU build based on your favorite registries or endpoint providers
- a bunch of commands to update and test your local data and the endpoints
- knows about aliases, so if you are tired of typing polakdot , dot will do as well
- can tap into the registry automatically generated by the Polkadot Network Directory

subrpc makes it easy to find an endpoint but most importantly, it helps building locally a list that is tailored to you and built up from decentralized sources that each user can freely define.

subrpc also makes it simple to share the list of your internal/private/testing endpoints.

Sample use

The following is an extract of the readme:

We first install subrpc, it is written in Rust so we use cargo:

cargo install subrpc

and check that it works:

subrpc --help
`subrpc` allows managing a set of registry providing rpc nodes

Usage: subrpc <COMMAND>

Commands:
  registry   Manage your registries
  system     System
  endpoints  Endpoints
  config     Config
  help       Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version

We can then make it aware of one or more registries using the subrpc registry add <url> command

# `reg` is an alias for `registry`
subrpc reg add https://raw.githubusercontent.com/chevdor/subrpc/master/registry/subrpc.json
subrpc reg add https://paritytech.github.io/polkadot_network_directory/registry.json
subrpc reg add https://raw.githubusercontent.com/chevdor/subrpc/master/registry/parity.json

It is now time to refresh the registry data:

subrpc reg up
subrpc system info

The output will look like:

Running subrpc v0.0.3
chevdor <chevdor@gmail.com>
local data file: /Users/will/.subrpc/data.json
- [ ] SubRPC Gist 1 - https://gist.githubusercontent.com/chevdor/a8b381911c28f6de02dde62ed1a17dec/raw/6992b0a2924f80f691e4844c1731564f0e2a62ec/data.json
      rpc endpoints: 2
      last update: None
- [X] Polkadot Network Directory - https://paritytech.github.io/polkadot_network_directory/registry.json
      rpc endpoints: 31
      last update: None
- [X] OnFinality - https://raw.githubusercontent.com/chevdor/subrpc/master/registry/dwellir.json
      rpc endpoints: 1
      last update: None
- [X] SubRPC Registry - https://raw.githubusercontent.com/chevdor/subrpc/master/registry/subrpc.json
      rpc endpoints: 3
      last update: None
- [X] Parity - https://raw.githubusercontent.com/chevdor/subrpc/master/registry/parity.json
      rpc endpoints: 13
      last update: None
- [X] Moonbeam Fundation - https://raw.githubusercontent.com/chevdor/subrpc/master/registry/moonbeam.json
      rpc endpoints: 3
      last update: None
- [ ] SubRPC Gist 2 - https://gist.githubusercontent.com/chevdor/a8b381911c28f6de02dde62ed1a17dec/raw/6992b0a2924f80f691e4844c1731564f0e2a62ec/data2.json
      rpc endpoints: 0
      last update: None

You may now tap in the list of known endpoints:

# `ep` is an alias for `endpoints`
subrpc ep ls

it will show the list of all known endpoints. The list is already quite long, you can test on your machine

In general, you will only be interested in one specific chain at a time though, this is what the subrpc endpoint get command does:

subrpc ep get polkadot

will output:

wss://polkadot.api.onfinality.io:443/public-ws
wss://rpc.polkadot.io:443
wss://polkadot-rpc.dwellir.com:443

Those are only http or web socket urls, but do they work ? subrpc endpoint ping will tell you:

RUST_LOG=debug subrpc ep ping

and I am planning some improvements on that front…

Call to infrastructure providers

If you make public (or private) endpoints available to the community, you could of course start with a PR and submit a file looking like this example.

It would be however even better if you host your own registry on your own server and share the link to your registry as a PR to the subrpc repository.

Even even better if you make those registraies also available in IPFS

Disclaimer: WIP & Early version

I decided to share this tool in its early state as I think it can be useful already adn trigger ideas and discussions. It is however definitely rough on the edges The data, commands and args will more than likely change over time.

Feedback, PRs and especially new data are welcome in the repository.

chevdor · January 27, 2023, 11:18am

If you end up juggling between several (para)chains in PolkadotJS, I published an update for subrpc that helps save quick some clicks and time ( terminal required):

install subrpc: cargo install subrpc and follow the quick start to have some data locally
source this function (it assumes you have fzf already installed… which you really should have anyway :))
you can then do this which opens the chain of your choice from your terminal without having to figure out whether your (para)chain is under Polkadot/Kusama or another of the category and without having to scroll all the way back up to “switch”

sourabhniyogi · February 14, 2023, 12:44am

This is great – some general questions:

Basic question: why is the list of RPC endpoints not on one specific common good chain (if not the relay chain)? It seems kind of crazy that everyone is using github as a CDN as the “off-chain” decentralized database for RPC endpoints. If you can store “identities” on the relay chain, why not RPC endpoints? Almost all the above questions apply to bootnodes, which parachain teams don’t always publish either, making running nodes more challenging.
For our indexing efforts, we have been using polkadot.js apps endpoints here since parachains almost always update first and is thus generally comprehensive – are you seeing RPC endpoints outside this list? (We could not find anything new … yet) We are specifically looking for RPC with ARCHIVE properties, then with “subscribeStorage” support. At the end of the day, we don’t want just a giant list of wss:// endpoints, we need the 1-3 archive endpoints that are reliable and have these specific features.
Can you make subrpc aggregated results flow back into polkadot.js apps on a regular basis? We would very much like the parachains to not have to update N places and for us to look in N places, thus question (1).

chevdor · February 14, 2023, 8:36am

Thanks for your feedback.

Basic question: why is the list of RPC endpoints not on one specific common good chain

While I agree with you that it would be nice not having all this data only in Github, I don’t think a (*)chain is the right place for that. First you would need the rpc endpoint to fetch… the rpc endpoints, which is not a great start.

I think it is fair for people to get started with hosting on Github or whatever option makes it simpler for them. A simple web hosting does the trick as well. Once registries are established, I would however love to see them pushed as well to decentralized services such as IPFS.

For our indexing efforts,…

As far as filtering endpoint, I am with you, this is a missing feature and it will come soon.
You may have seen that the schema offers the ability to store labels at the provider level and also at the endpoint level. I invite registries to start adding labels to their data (parity already started) so users can use that feature soon.

Users of the lib could code the filtering themselves already today. I plan however to add tihs both to the lib and to the cli so that you can specifically --include or --exclude labels such as archive or inrternal for instance. I could also imagine labels such as aws, gcp, etc… to specifiy (when known) where the nodes run and allow users to specify whether they want to include/exclude nodes on a specific platform.

Can you make subrpc aggregated results flow back into polkadot.js

The discussion was already mentioned here. I see no reason not to do it once we have good data but I don’t think we are there yet.

Ultimately, the decision is in Jaco’s hand and I would not suggest the switch yet as I don’t think we have the full data yet. Once we do, and also have the filtering feature, I am sure it will make updates of the list of endpoints in PolkadotJS much easier.

As an alternate step, PolkadotJS could transfer their data to a publicly available registry and use this its very own data (augmented or not with other registries) to feedback into the generated code providing the list of available endpoints today.

chevdor · March 3, 2023, 9:48am

I just added a bunch of endopoints, so you may benefit from making sure you did:

Add all available registries (see the list at GitHub - chevdor/subrpc)
run a subrpc registry update to fetch all the latest data

Topic		Replies	Views
Polkadot Network Directory	1	532	December 13, 2022
A new test network for Polkadot Digest	17	1509	October 24, 2023
ZombieNet registry Tech Talk zombienet	2	471	September 8, 2022
Scaling Down of Parity’s Public Infrastructure Ecosystem	5	1147	November 29, 2023
Test frameworks and utilities for Polkadot and friends Tech Talk	3	619	April 17, 2023