# \`subrpc\`: manage multiple endpoint registries and quickly find endpoints

**URL:** <https://forum.polkadot.network/t/subrpc-manage-multiple-endpoint-registries-and-quickly-find-endpoints/1748>\
**Category:** Tech Talk\
**Created:** [January 20, 2023, 11:29am UTC](https://forum.polkadot.network/t/subrpc-manage-multiple-endpoint-registries-and-quickly-find-endpoints/1748 "2023-01-20T11:29:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chevdor](https://dub1.discourse-cdn.com/flex005/user_avatar/forum.polkadot.network/chevdor/32/217_2.png) [@chevdor](https://forum.polkadot.network/u/chevdor)\
**Post date:** [January 20, 2023, 11:29am UTC](https://forum.polkadot.network/t/subrpc-manage-multiple-endpoint-registries-and-quickly-find-endpoints/1748/1 "2023-01-20T11:29:17Z")

</div>

## `subrpc`

I am happy to share a new cli tool: `subrpc`.

## Context

I recently posted about the [Polkadot Network Directory](https://forum.polkadot.network/t/polkadot-network-directory/1376) which helps find chains and informations related chains and parachains.

Here is the repo:

> **[GitHub - paritytech/polkadot\_network\_directory](https://github.com/paritytech/polkadot_network_directory)**
>
> Contribute to paritytech/polkadot\_network\_directory development by creating an account on GitHub.

Many projects [already contributed](https://github.com/paritytech/polkadot_network_directory/pulls?q=is%3Apr+is%3Aclosed) with PRs bringing their data to the directory.

One of the benefits of having this data is that we could also generate a registry: a list of available endpoints:  
[https://paritytech.github.io/polkadot\_network\_directory/registry.json](https://paritytech.github.io/polkadot_network_directory/registry.json)

There is however no reason for this data to be centralized in one location and users may be interested in either subsets or augmented data set.

I am happy to share the first versions of [`subrpc`](https://github.com/chevdor/subrpc). You can check its [readme](https://github.com/chevdor/subrpc#readme) for the full details.

## TLDR

The TLDR is:

- `subrpc` is a:
  - cli written in Rust you can use in your terminal to find endpoints: `subrpc ep get dot`
  - a lib/crate you can use from your app to connect to the right endpoint without having to manage that part in depth
  - local database YOU build based on your favorite registries or endpoint providers
  - a bunch of commands to update and test your local data and the endpoints
  - knows about aliases, so if you are tired of typing `polakdot` 🙂 , `dot` will do as well
  - can tap into the registry automatically generated by the [Polkadot Network Directory](https://paritytech.github.io/polkadot_network_directory/)

`subrpc` makes it easy to find an endpoint but most importantly, it helps building locally a list that is tailored to you and built up from **decentralized sources** that each user can freely define.

`subrpc` also makes it simple to share the list of your internal/private/testing endpoints.

## Sample use

The following is an extract of the [readme](https://github.com/chevdor/subrpc#readme):

We first install `subrpc`, it is written in Rust so we use `cargo`:

```auto
cargo install subrpc

```

and check that it works:

```auto
subrpc --help
`subrpc` allows managing a set of registry providing rpc nodes

Usage: subrpc <COMMAND>

Commands:
  registry Manage your registries
  system System
  endpoints Endpoints
  config Config
  help Print this message or the help of the given subcommand(s)

Options:
  -h, --help Print help
  -V, --version Print version

```

We can then make it aware of one or more registries using the `subrpc registry add <url>` command

```auto
# `reg` is an alias for `registry`
subrpc reg add https://raw.githubusercontent.com/chevdor/subrpc/master/registry/subrpc.json
subrpc reg add https://paritytech.github.io/polkadot_network_directory/registry.json
subrpc reg add https://raw.githubusercontent.com/chevdor/subrpc/master/registry/parity.json

```

It is now time to refresh the registry data:

```auto
subrpc reg up
subrpc system info

```

The output will look like:

```auto
Running subrpc v0.0.3
chevdor <chevdor@gmail.com>
local data file: /Users/will/.subrpc/data.json
- [] SubRPC Gist 1 - https://gist.githubusercontent.com/chevdor/a8b381911c28f6de02dde62ed1a17dec/raw/6992b0a2924f80f691e4844c1731564f0e2a62ec/data.json
      rpc endpoints: 2
      last update: None
- [X] Polkadot Network Directory - https://paritytech.github.io/polkadot_network_directory/registry.json
      rpc endpoints: 31
      last update: None
- [X] OnFinality - https://raw.githubusercontent.com/chevdor/subrpc/master/registry/dwellir.json
      rpc endpoints: 1
      last update: None
- [X] SubRPC Registry - https://raw.githubusercontent.com/chevdor/subrpc/master/registry/subrpc.json
      rpc endpoints: 3
      last update: None
- [X] Parity - https://raw.githubusercontent.com/chevdor/subrpc/master/registry/parity.json
      rpc endpoints: 13
      last update: None
- [X] Moonbeam Fundation - https://raw.githubusercontent.com/chevdor/subrpc/master/registry/moonbeam.json
      rpc endpoints: 3
      last update: None
- [] SubRPC Gist 2 - https://gist.githubusercontent.com/chevdor/a8b381911c28f6de02dde62ed1a17dec/raw/6992b0a2924f80f691e4844c1731564f0e2a62ec/data2.json
      rpc endpoints: 0
      last update: None

```

You may now tap in the list of known endpoints:

```auto
# `ep` is an alias for `endpoints`
subrpc ep ls

```

it will show the list of all known endpoints. The list is already quite long, you can test on your machine 🙂

In general, you will only be interested in one specific chain at a time though, this is what the `subrpc endpoint get` command does:

```auto
subrpc ep get polkadot

```

will output:

```auto
wss://polkadot.api.onfinality.io:443/public-ws
wss://rpc.polkadot.io:443
wss://polkadot-rpc.dwellir.com:443

```

Those are only http or web socket urls, but do they work ? `subrpc endpoint ping` will tell you:

```auto
RUST_LOG=debug subrpc ep ping

```

and I am planning some improvements on that front…

## Call to infrastructure providers

If you make public (or private) endpoints available to the community, you could of course start with a PR and submit a file looking like [this example](https://github.com/chevdor/subrpc/blob/eec6201a51a1c27e9f46c206a1e06ab3205317f5/registry/parity.json).

It would be however even better if you host your own registry on your own server and share the link to your registry as a PR to the `subrpc` repository.

Even even better if you make those registraies also available in IPFS 🙂

## Disclaimer: WIP & Early version

I decided to share this tool in its early state as I think it can be useful already adn trigger ideas and discussions. It is however definitely rough on the edges 🙂 The data, commands and args will more than likely change over time.

Feedback, PRs and especially new data are welcome in the [repository](https://github.com/chevdor/subrpc).

---

<div class="post-metadata">

**Author:** ![chevdor](https://dub1.discourse-cdn.com/flex005/user_avatar/forum.polkadot.network/chevdor/32/217_2.png) [@chevdor](https://forum.polkadot.network/u/chevdor)\
**Post date:** [January 27, 2023, 11:18am UTC](https://forum.polkadot.network/t/subrpc-manage-multiple-endpoint-registries-and-quickly-find-endpoints/1748/2 "2023-01-27T11:18:39Z")

</div>

If you end up juggling between several (para)chains in PolkadotJS, I published an update for `subrpc` that helps save quick some clicks and time (🧑‍💻 terminal required):

- install `subrpc`: `cargo install subrpc` and follow the [quick start](https://github.com/chevdor/subrpc#quick-start) to have some data locally
- source [this function](https://github.com/chevdor/subrpc/blob/70e33e1868fe5797c056b19c21e13c745804a12c/doc/subopen.sh) (it assumes you have [fzf](https://github.com/junegunn/fzf) already installed… which you really should have anyway :))
- you can then do [this](https://asciinema.org/a/7R4qTXsZmnF5kuH6sC2jveY8W) which opens the chain of your choice from your terminal without having to figure out whether your (para)chain is under Polkadot/Kusama or another of the category and without having to scroll all the way back up to “switch”

---

<div class="post-metadata">

**Author:** ![sourabhniyogi](https://avatars.discourse-cdn.com/v4/letter/s/35a633/32.png) [@sourabhniyogi](https://forum.polkadot.network/u/sourabhniyogi)\
**Post date:** [February 14, 2023, 12:44am UTC](https://forum.polkadot.network/t/subrpc-manage-multiple-endpoint-registries-and-quickly-find-endpoints/1748/3 "2023-02-14T00:44:37Z")

</div>

This is great – some general questions:

1. Basic question: why is the list of RPC endpoints not on one specific common good chain (if not the relay chain)? It seems kind of crazy that everyone is using github as a CDN as the “off-chain” decentralized database for RPC endpoints. If you can store “identities” on the relay chain, why not RPC endpoints? Almost all the above questions apply to bootnodes, which parachain teams don’t always publish either, making running nodes more challenging.

2. For our indexing efforts, we have been using polkadot.js apps endpoints [here](https://github.com/polkadot-js/apps/blob/9d191f54a8273013c8c15fecde93d604600cefcc/packages/apps-config/src/endpoints/productionRelayPolkadot.ts) since parachains almost always update first and is thus generally comprehensive – are you seeing RPC endpoints outside this list? (We could not find anything new … yet) We are specifically looking for RPC with _ARCHIVE_ properties, then with “subscribeStorage” support. At the end of the day, we don’t want just a giant list of wss:// endpoints, we need the 1-3 archive endpoints that are reliable and have these specific features.

3. Can you make subrpc aggregated results flow back into polkadot.js apps on a regular basis? We would very much like the parachains to not have to update N places and for us to look in N places, thus question (1).

---

<div class="post-metadata">

**Author:** ![chevdor](https://dub1.discourse-cdn.com/flex005/user_avatar/forum.polkadot.network/chevdor/32/217_2.png) [@chevdor](https://forum.polkadot.network/u/chevdor)\
**Post date:** [February 14, 2023, 8:36am UTC](https://forum.polkadot.network/t/subrpc-manage-multiple-endpoint-registries-and-quickly-find-endpoints/1748/4 "2023-02-14T08:36:02Z")

</div>

Thanks for your feedback.

> 1. Basic question: why is the list of RPC endpoints not on one specific common good chain

While I agree with you that it would be nice not having all this data only in Github, I don’t think a (\*)chain is the right place for that. First you would need the rpc endpoint to fetch… the rpc endpoints, which is not a great start.

I think it is fair for people to get started with hosting on Github or whatever option makes it simpler for them. A simple web hosting does the trick as well. Once registries are established, I would however love to see them pushed as well to decentralized services such as IPFS.

> 1. For our indexing efforts,…

As far as **filtering** endpoint, I am with you, this is a missing feature and it will come soon.  
You may have seen that the schema offers the ability to store `labels` at the provider level and also at the endpoint level. I invite registries to start adding labels to their data (parity already started) so users can use that feature soon.

Users of the lib could code the filtering themselves already today. I plan however to add tihs both to the lib and to the cli so that you can specifically `--include` or `--exclude` labels such as `archive` or `inrternal` for instance. I could also imagine labels such as `aws`, `gcp`, etc… to specifiy (when known) where the nodes run and allow users to specify whether they want to include/exclude nodes on a specific platform.

> 1. Can you make subrpc aggregated results flow back into polkadot.js

The discussion was already mentioned [here](https://github.com/paritytech/polkadot_network_directory/issues/23). I see no reason not to do it once we have good data but I don’t think we are there yet.

Ultimately, the decision is in Jaco’s hand and I would not suggest the switch yet as I don’t think we have the full data yet. Once we do, and also have the filtering feature, I am sure it will make updates of the list of endpoints in PolkadotJS much easier.

As an alternate step, PolkadotJS could transfer their data to a publicly available registry and use this its very own data (augmented or not with other registries) to feedback into the generated code providing the list of available endpoints today.

---

<div class="post-metadata">

**Author:** ![chevdor](https://dub1.discourse-cdn.com/flex005/user_avatar/forum.polkadot.network/chevdor/32/217_2.png) [@chevdor](https://forum.polkadot.network/u/chevdor)\
**Post date:** [March 3, 2023, 9:48am UTC](https://forum.polkadot.network/t/subrpc-manage-multiple-endpoint-registries-and-quickly-find-endpoints/1748/5 "2023-03-03T09:48:46Z")

</div>

I just added a bunch of endopoints, so you may benefit from making sure you did:

- Add all available registries (see the list at [GitHub - chevdor/subrpc](https://github.com/chevdor/subrpc#quick-start))
- run a `subrpc registry update` to fetch all the latest data