This is an extension to Stabilizing Polkadot - #15 by OliverTY, specifically the ideas around Omni-Node.
Problem?
The idea of the omni-node stems from the fact that for many builders, maintaining the node-side software is mere overhead with little benefit. If a team doesn’t want to customize the node components and simply follow the defaults, maintaining the node is not all that useful.
This is why we have heavily relied so far on “templates” in our ecosystem, as this node boilerplate had to be handed over to developers in some way.
But, even with proper perfect semver-aware releases, and perfect templates, there are still issues with maintaining a node:
- Figuring out the right version to update to.
For this purpose, @OliverTY has suggested an umbrella crate, and in the meantime psvm is a great tool to help builders.
- Figuring out the right changes that need to be applied.
For example, to enable PoV reclaim or async-backing, both of which are important features that we expect all parachain teams need to incorporate, some degree of change was needed in the node-side code. This usually includes a parachain developer to look into a correct example such as the
polkadot-parachain-bincrate maintained by parity to run system parachain collators, and copy-paste the changes.
Lastly, as polkadot-sdk is still working toward removing the native runtime from the node, it only makes sense to have more omni-nodes rather than custom nodes.
Solution
The “omni-node” idea consists of two steps to solve this:
Goal-0: Provide a single binary that can sync all parachains so long as they adhere to some standards. We can foresee a few ways to run this omni-node, in the order of complexity respectively:
./omni-node --chain parachain.json -- --chain relay.json
# Maybe possible: get the runtime and genesis state from relay-chain, or elsewhere
./omni-node --para-id 2345
As noted below, the
polkadot-parachainbinary is already almost exactly what we want here.
In general, there is no limit to how much further we can bloat this binary. By bloating, I mean adding more features to it, such that a broader group of teams can use it without needing to compile anything. Some ideas here are:
- EVM/Frontier/Contract support
- Further consensus types
- Custom RPCs
- Ability to be both a solochain and a parachain
- Custom host functions
All of this will come at some cost, in that the code of the omni-node will become bigger and bigger, and harder and harder to maintain.
Goal-1: A node-builder crate to allow compilation of custom nodes with smaller code footprint.
Through building Goal-0, we can possibly clean up the interfaces needed to build the node side into a more reasonable level, such that we can foresee the following as the effort needed to run a node:
use polkadot_sdk_node::Builder as NodeBuider;
// A standard example
fn main() -> sc_cli::Result<()> {
let node = NodeBuilder::default()
.node_type(builder::NodeType::Parachain)
.authoring(builder::Authoring::Aura)
.finality(builder::Finality::None)
.extra_rpc(frame_system::Rpc)
.extra_rpc(frame_transaction_payment_rpc::Rpc)
.build()
.unwrap();
node.run();
}
// An ethereum enabled parachain
fn main() -> sc_cli::Result<()> {
let node = NodeBuilder::default()
.node_type(builder::NodeType::Parachain)
.frontier() // sets all the right configs
.build()
.unwrap();
node.run();
}
// a PoW solochain node.
fn main() -> sc_cli::Result<()> {
let node = NodeBuilder::default()
.node_type(builder::NodeType::Solochain)
.finality(builder::Finality::None)
.authoring(builder::Authoring::PoW)
.build()
.unwrap();
node.run();
}
This would mean that the aforementioned omni-nodes are merely pre-compiled instance of what the polkadot-sdk-node-builder crate already provides. A team would then have the option to use one of the pre-compiled versions, or compile their own node using the builder crate.
Personal opinion: As stated here, I think building the node-builder is the right approach, as it forces us to repay some of the technical debt and structure the service/node side code into a cleaner software artifact.
Existing polkadot-parachain
The good news here is that the existing polkadot-parachain binary maintained in polkadot-sdk is already to a high extent decoupled from the system parachain runtimes and is therefore is an omni-node-beta. This node will by default spin up an aura-based parachain node, and is therefore well capable of syncing and hypothetically collating on any parachain that has a similar assumption.
The general usage pattern that I have tried is as follows:
./polkadot-parachain --chain ./specs/parachain.json -- --chain ./specs/relay.json --sync warp
Questions
- What are some examples of parachain teams needing to customize the node side, which I may have missed so far? What I know so far, as mentioned above:
- Runtime types, such as
AccountId,Hash, or evenBlocktype. In cases such as a frontier-enabled parachain, these types might differ. - EVM compatible parachains.
- Using different consensus. For example, many system chains start with the “no-consensus” code, and graduate to “aura”. I know some teams use Nimbus from moonbeam.
- Custom RPCs. Custom RPCs are something that we consider by now “bad practice” (in favor of using custom runtime-api +
state_call), as @xlc also pointed out another alternative here. - Custom host functions
- Custom Database (EVM Database)
What else?
- What are your common development/DevOps scenarios where you would have to maintain a node now, but ideally you want to get rid of it? For example, it would be great if developing a parachain, especially in the world of agile-coretime, can be entirely done using
chain-spec-builder+ a runtime/.wasmfile.
Next
- make the minimum changes to
polkadot-parachainsuch that it can run many parachains, publish it and encourage teams to use it for feedback. - Based on the feedback and need, remove more assumptions from it and make it more flexible/comprehensive, as per `parachain-omni-node` binary by kianenigma · Pull Request #3597 · paritytech/polkadot-sdk · GitHub
Related Resources
- Vision for cleaner substrate-node's developer-facing interface · Issue #5 · paritytech/polkadot-sdk · GitHub
- Fix `polkadot-parachain` to work just with chain-spec · Issue #3944 · paritytech/polkadot-sdk · GitHub
- `parachain-omni-node` binary by kianenigma · Pull Request #3597 · paritytech/polkadot-sdk · GitHub
- [DNM] SDK umbrella crate by ggwpez · Pull Request #3935 · paritytech/polkadot-sdk · GitHub
- release chain-spec-builder · Issue #4352 · paritytech/polkadot-sdk · GitHub