Polkadot Parachain Omni-Node: Gathering Ideas and Feedback

This is an extension to Stabilizing Polkadot - #15 by OliverTY, specifically the ideas around Omni-Node.

Problem?

The idea of the omni-node stems from the fact that for many builders, maintaining the node-side software is mere overhead with little benefit. If a team doesn’t want to customize the node components and simply follow the defaults, maintaining the node is not all that useful.

This is why we have heavily relied so far on “templates” in our ecosystem, as this node boilerplate had to be handed over to developers in some way.

But, even with proper perfect semver-aware releases, and perfect templates, there are still issues with maintaining a node:

  1. Figuring out the right version to update to.

For this purpose, @OliverTY has suggested an umbrella crate, and in the meantime psvm is a great tool to help builders.

  1. Figuring out the right changes that need to be applied.

For example, to enable PoV reclaim or async-backing, both of which are important features that we expect all parachain teams need to incorporate, some degree of change was needed in the node-side code. This usually includes a parachain developer to look into a correct example such as the polkadot-parachain-bin crate maintained by parity to run system parachain collators, and copy-paste the changes.

Lastly, as polkadot-sdk is still working toward removing the native runtime from the node, it only makes sense to have more omni-nodes rather than custom nodes.

Solution

The “omni-node” idea consists of two steps to solve this:

Goal-0: Provide a single binary that can sync all parachains so long as they adhere to some standards. We can foresee a few ways to run this omni-node, in the order of complexity respectively:

./omni-node --chain parachain.json -- --chain relay.json

# Maybe possible: get the runtime and genesis state from relay-chain, or elsewhere
./omni-node --para-id 2345

As noted below, the polkadot-parachain binary is already almost exactly what we want here.

In general, there is no limit to how much further we can bloat this binary. By bloating, I mean adding more features to it, such that a broader group of teams can use it without needing to compile anything. Some ideas here are:

  1. EVM/Frontier/Contract support
  2. Further consensus types
  3. Custom RPCs
  4. Ability to be both a solochain and a parachain
  5. Custom host functions

All of this will come at some cost, in that the code of the omni-node will become bigger and bigger, and harder and harder to maintain.

Goal-1: A node-builder crate to allow compilation of custom nodes with smaller code footprint.

Through building Goal-0, we can possibly clean up the interfaces needed to build the node side into a more reasonable level, such that we can foresee the following as the effort needed to run a node:

use polkadot_sdk_node::Builder as NodeBuider; 

// A standard example
fn main() -> sc_cli::Result<()> {
	let node = NodeBuilder::default()
		.node_type(builder::NodeType::Parachain)
		.authoring(builder::Authoring::Aura)
		.finality(builder::Finality::None)
		.extra_rpc(frame_system::Rpc)
		.extra_rpc(frame_transaction_payment_rpc::Rpc)
		.build()
		.unwrap();
	node.run();
}

// An ethereum enabled parachain 
fn main() -> sc_cli::Result<()> {
	let node = NodeBuilder::default()
		.node_type(builder::NodeType::Parachain)
		.frontier() // sets all the right configs
		.build()
		.unwrap();
	node.run();
} 

// a PoW solochain node. 
fn main() -> sc_cli::Result<()> {
	let node = NodeBuilder::default()
		.node_type(builder::NodeType::Solochain)
		.finality(builder::Finality::None)
		.authoring(builder::Authoring::PoW)
		.build()
		.unwrap();
	node.run();
} 

This would mean that the aforementioned omni-nodes are merely pre-compiled instance of what the polkadot-sdk-node-builder crate already provides. A team would then have the option to use one of the pre-compiled versions, or compile their own node using the builder crate.

Personal opinion: As stated here, I think building the node-builder is the right approach, as it forces us to repay some of the technical debt and structure the service/node side code into a cleaner software artifact.

Existing polkadot-parachain

The good news here is that the existing polkadot-parachain binary maintained in polkadot-sdk is already to a high extent decoupled from the system parachain runtimes and is therefore is an omni-node-beta. This node will by default spin up an aura-based parachain node, and is therefore well capable of syncing and hypothetically collating on any parachain that has a similar assumption.

The general usage pattern that I have tried is as follows:

./polkadot-parachain --chain ./specs/parachain.json -- --chain ./specs/relay.json --sync warp

Questions

  1. What are some examples of parachain teams needing to customize the node side, which I may have missed so far? What I know so far, as mentioned above:
  • Runtime types, such as AccountId, Hash, or even Block type. In cases such as a frontier-enabled parachain, these types might differ.
  • EVM compatible parachains.
  • Using different consensus. For example, many system chains start with the “no-consensus” code, and graduate to “aura”. I know some teams use Nimbus from moonbeam.
  • Custom RPCs. Custom RPCs are something that we consider by now “bad practice” (in favor of using custom runtime-api + state_call), as @xlc also pointed out another alternative here.
  • Custom host functions
  • Custom Database (EVM Database)

What else?

  1. What are your common development/DevOps scenarios where you would have to maintain a node now, but ideally you want to get rid of it? For example, it would be great if developing a parachain, especially in the world of agile-coretime, can be entirely done using chain-spec-builder + a runtime/.wasm file.

Next

Related Resources

12 Likes

I have created 3 short videos where I explain this forum post, and showcase some examples:

10 Likes

I like the idea of a well abstracted node builder, a similar concept that was a nice experience was a custom reverse proxy I had to make with Cloudflare’s pingora, a few lines of code in the main function was enough to have a powerful custom proxy. If the builder can do something similar where you just need a couple of lines to have a node with good defaults that would be great(extending it and adding some extra bits should still be simple).

2 Likes

Great post - will definitely reduce friction in spinning up tasks!

Pleased to say that I was able to drop-in polkadot-parachain as a replacement node implementation for the parachain template!

My steps were as follows:

  1. Compiled polkadot-parachain with the same version as my runtime code (v1.10.0)
  2. Use my parachain-template-node to generate my chain-spec (probably the one thing I still had to use the binary for)
  3. Passed in my chain spec and OG sync settings for Rococo, along with a base path, and it synced with no issues. Was able to issue an on-demand extrinsic and collate:
./target/release/polkadot-parachain --alice \
--collator \
--chain ./specs/4418.json \
--base-path /Volumes/My\ Passport/ \
--force-authoring \
-- \
--chain rococo \
--sync fast-unsafe \
--blocks-pruning 256

One question - where would the generation for genesis / chain spec CLI functionality live? Could it be that this is a separate tool, or that creating a chain_spec.rs is part of the runtime side of development?

Thanks for trying this out Bader!

You can use the chain-spec-builder independent binary to generate the chain spec from a wasm file.

At the moment it is not released, but it soon will be:

Example:

At the moment the chain spec builder can generate either default chain specs, and/or patch them with a json file. @michal is working on a new system to have “presets” of chain-specs being generated from within the runtime. This will extend the default variant and remove the need for a patch file.

5 Likes

Given this is a node that, it would appear, is going to be widely used is it possible to clarify and address where this node sits with respect to the Polkadot Patent, see here:

Specifically, at a minimum can a clear statement be made upfront in the documentation about the extent to which using such a node is infringing the patent.

Given this patent is granted through to late 2038, there is some sense making head room for things that might only emerge over time. A couple of ideas:

  1. Clearly state in the docs where this node sits wrt to the patent. Example: Is it possible to use the node in a configuration that is patent free?
  2. If it is maybe add a "patent" feature now, even if it is effectively a no-op. And everything is patent free unless it is moved under that feature flag - this way Parity has to state explicitly what they consider to be covered by the patent. Or maybe you are told everything has to be moved under the "patent" feature flag, and running the node without the "patent" feature flag results in crickets/no-op. At least users would now be aware of what they are doing and don’t inadvertently violate Parity’s rights.

Hopefully that is clear. At the least, this should remove ambiguity, which is where the precautionary principle tends to lead people to be more averse than might be warranted.

Both solutions have pros/cons but, in my opinion, having the builder which could offer different level of granularity could be a good way to solve this problem and also provide a better approach to understand/tweak a substrate node over time.

The challenge with the simplified builder is that upgrading will bring undetected changes (some of them breaking components compatiblity) which might be “ok” in a dev environment but could be problematic otherwise.

I like the idea of separating the runtime from the node either way. Runtime versions and node versions are too closely linked in Frequency and I think a decoupled setup would have been the choice had it been easy to do when it was started.

What are some examples of parachain teams needing to customize the node side?

The Frequency Parachain has done a few node level modifications:

Custom RPCs: We have one or two that I am not convinced a runtime-api + state call is truly able to replace.

Custom RPCs are great in a few situations, which I admit are uncommon:

  • The code to process the call is heavier than would be desired inside of the runtime
  • The processing is better when native (becoming less of a concern as performance improves)

Frequency also has an off-chain indexer. It handles generating a reverse index of data that is available on chain (A → B), but often users want/need the other direction (B → A). There is no reason to store that information inside consensus of course, and while a sidecar service can perform that action, it is nice to have it packaged together.

Customization ideas we’ve had that we have not done (and might be bad ideas):

  • Altering the storage system so that some portions of the historical state are not retained by default (although keeping the root hash).
  • Providing additional integration with ipfs. Like as any extrinsic that takes a CID is able to check availability of that CID before continuing to post the transaction. Or able to retrieve IPFS data along with the event.

With the exception of the storage idea (which might well have to be entirely inside the runtime anyway), I don’t think it invalidates having an omni-node for most use cases. Assuming no modifications to the node are operational, things like custom RPCs and such could still be available via a custom node while many/most node operators use an omni-node.

Yeah, this is the problem with abstraction as usual. I would argue that teams that can afford the cost should maintain a full node software crate, and in return they will remain informed of more such changes.

Both of these are valid arguments, but you should be aware that both basically break the assumption of forkelss upgrade in some cases. in that, those RPCs that you put inside the Node/Native side will either have to be static, or else their upgrade process will be difficult as they are not part of the WASM blob.

Have you considered @xlc’s suggestion as noted here?

Can you point me to which of your RPCs are entirely handled within the node side?

Indeed, I find your examples interesting use-cases, but it is arguably fair to expect teams that want to do some extensive customization to use a full node template code base, so all of that would be outside the scope of the omni-node.

We’ve been moving many things in that direction. We’re working on a group of job specific sidecars which mostly follows that same pattern: GitHub - AmplicaLabs/gateway: Gateway microservices for DSNP/Frequency

The get_messages_by_schema_id is the one that is the most confusing for the blockchain side of things as Frequency stores messages in state instead of at the block level (even though the data is block-centric) so that you don’t need to run an archive node to be able to access that set of data. (I have the goal of removing that data from archive nodes as well as you might have guessed from the idea list).

If needed, all of those could be moved into sidecars and such.

Most of the cases resolve around the question of how can we make it easier to use the chain. Once you get outside of JavaScript, it is not easy to integrate. JS based sidecars solve some of that issue, but leave a lot to be desired for performance.

First of all, I think that is a great development into the right direction! Will make it much easier for externals to integrate parachains into their stack.

For us, if this is supported, we are happy. The only thing I could think of was custom inherents, but then our validators just need to run our custom node instead of the omni-node. Which I think is a reasonable compromise.

1 Like