RFC: Should we launch a Thousand Cores program?

Governance
1

Summary

With core time sales, agile core time and elastic scaling Polkadot will need to scale to >1000 apps built on it. Our infrastructure (block explorers, collators etc.) aren’t ready for this 10x increase from the current number of parachains. Let’s start an initiative to address these issues.

Introduction & Motivation

The original vision of Polkadot was to have parachains. People built architecture around single chains and they were always seen as relatively big standalone projects. As a result a lot of the infrastructure for running parachains built early on doesn’t actually scale to the thousands of apps that we want to see launch on Polkadot. The base layer is making great progress towards simplifying development and deployment: replacing auctions with core time sales, agile coretime (fka parathreads) and elastic scaling drastically lower the barrier of entry for new entrants to get started with Polkadot. However this doesn’t expand beyond the core service Polkadot offers; network security. When launching an application projects need to deal with different service providers that all have manual work and significant cost to support new apps offering too much customization where it’s not needed and not enough convenience.

What’s missing beyond the core protocol changes?

The Polkadot Fellowship and Parity with its new focus on protocol development have been pushing many of the protocol features forward that make Polkadot more scalable and easier to build on.

Parity also started an effort to build “the omni-node” creating a single node that will be able to run most parachains removing the need for custom binaries to be built and maintained. This will significantly reduce the effort for infrastructure providers to maintain different codebases and maintain more than one client in their infrastructure.

This however isn’t enough; to run a typcial parachain most projects rely on the following offchain services as well:

  • Hosted full nodes
  • Block explorers
  • Governance tooling
  • Wallets
  • Collators
  • On/off ramps, exchange integrations

Especially with the launch of agile core time the number of apps will grow massively and many of the above pieces of critical infrastructure providers aren’t ready to handle this.

Thousand Cores Program: focusing on closing these gaps

The Thousand Validators program in Polkadot was very successful in creating a diverse set validators improving the decentralization of Polkadot. As a nod to this success story, I want to propose we launch the Thousand Cores program as an initiative that aims to support efforts in tooling and infrastructure improvements that help us achieve the following goals:

  1. A Polkadot app developer must be able to launch a fully functioning chain with a distributed collator set, a functioning block explorer and other essential parts in a day and simply with self serve and automated tools.
  2. Our ecosystem must support thousands of cores with many of them coming and going easily without massive overhead.
  3. Launchig a core should be easier than launching an L2 and almost as easy as deploying a smart contract on an existing EVM chain.
  4. The all in cost of launching an app in Polkadot should be similar or less than launching an app chain, sovereign roll up, L2/L3.

What should the initiative do? How do we achieve this goal? What’s next?

The primary goal of the initiative is to draw attention to the gaps we have and make sure we fix them proactively and bring people together to brainstorm and collaborate on these issues.

The DeFi and Infrastructure ecosystem bounty has budget to address some of these hurdles and I would like to hear from builders what the current biggest hurdles are and what tooling (e.g. block explorers, gov tooling, wallets, analytics tools, collator providers).

There are a lot of initiatives already that work towards this goal.
I would propose that we start asking all infrastructure provider to make proposals on how they can help us contribute in this goal and what the missing resources are (omni-node, standards etc.). Where necessary we can use the bounty to fund infrastructure and where simply better coordination is needed, Velocity Labs could step in and help with coordination.

What do you think of this idea? What pieces do you see missing for us to realize this reality?

13 Likes
2

How do you see your proposal in relation to Tanssi. Afaik it intends to provide those services for devs.

1 Like
3

Tanssi is very much aligned in addressing these issues and would be one of the projects that should be supported with this initiative. However just supporting Tanssi isn’t enough. There are parts of the stack they can’t solve though which hopefully we can get more attention.

Beyond Tanssi, there are block explorers, governance tooling, full nodes and wallets that all aren’t well equipped that to handle thousands of apps that will be built on Polkadot.

2 Likes
4

There is seemingly nothing about “cores” here in the technical sense, so the name makes little sense.

Just fyi…

If we push the polkadot v cpu+os analogy, then parachains resemble processes, coretime sales resembles accounting & prioritization like acct & nice, and our “cores” resemble cpu cores, into which the relay chain schedules processes. We’ll have more total processing if we have more cores, but parachains looks pretty independent, like maybe many parachains make blocks only once every 10 minutes.

We must limit total code size across all parachains of course, for which the polkavm folks envision doing dynamic linking eventually. I’d kinda expect dynamic linking would play poorly with the current codebase, but we’re far away from these costs being problematic, we could transition piecemeal towards dynamic limnking friendly code, and spree imposes more constraints than dynamic linking does.

At present, we add cores promarily through optimizing performance, but we expect one relay chain of 1000 validators cannot go beyond 300-500 full speed cores. We could improve node specs too of course, see trade offs. We could add multiple relay chains too, by using an improved analysis to omniledger so their byzantine assumptions agree, but even this does not improve the code size limit per se.

I’d expect 1000 cores likely means 2000 validators across 2 relay chains, but maybe more if optimizations work out poorly, too many low spec validators hold too much voting power, etc.

Anyways…

None of this is what you wanted to discuss. I just wanted to point out the name fits poorly.

1 Like
5

I agree with the the goals of this program Lucas, thanks for taking the initiative. The friction that devs face when launching chains on polkadot was a primary motivation for Tanssi, which I’ve been involved with (born from trying to improve our experience with moonbeam). My goal has always been to see devs able to deploy a production chain on Polkadot with all of the key required integrations (wallets, bridges, indexers, explorers, etc) within minutes / one working session. That would be very powerful and unique in the market – L1 as a decentralized service. Tanssi covers core services such as block production and data retrieval, and has been working on many of these problems with other teams (e.g. Moonbeam Routed Liquidty as a baseline token bridge that could be available out of the box on a freshly deployed tanssi containerchain). I would welcome additional funding and support for automation around other key components. For example, permissionless Talisman wallet support, subsidies and automation for Subscan support, etc.

3 Likes
6

@burdges you’re right, this has been pointed out to me by someone else as well. Name change incoming. Between Kusama and Polkadot we’ll realistically only get to a couple hundred cores. That’s stil a lot more than what we have but perhaps we should call it the “Thousand Parachains” program.

8

Thanks for the Pop CLI mention Alex!

As Alex said, Pop CLI can help with this as well. Currently, Pop CLI does not cover the production deployment portion, but what it does do is make local testing and development quite easy.

We imagine that many people will want to deploy a parachain with “1-click”. However, we have also talked to builders who would rather test things locally – where they have the control – before deploying. This goes for parachain builders, but also other builders like smart contract devs and offchain applications (UIs, etc.).

This is what Pop CLI specializes in. Pop CLI will integrate the relevant templates / use-cases so somebody can easily setup a new project and start hacking. Long term, we want to integrate with infrastructure providers to bridge the gap between local development and deployment.

Always happy to talk more about it!

3 Likes
9

We wholeheartedly agree with this statement, and we think Polkadot needs to have a wider range of options for builders to deploy with just a few clicks. The landscape of the Ethereum ecosystem shows that 1-click deployment solutions are lately a go-to for builders, as they dramatically reduce the risk and complexity of deploying.

As mentioned earlier, however, these solutions have to come accompanied with a range of tools out-of-the-box for them to be truly successful. And Polkadot has unique propositions, like Coretime, that need to be carefully catered for.

We built a PoC before where anyone can, with three clicks, deploy a template and have a Parachain running. Once running, you can manage your Coretime needs, and even dry-run runtime upgrades with our Chopsticks integration. Thank you @alexdimes by the way for using our solution, we really appreciate it :heavy_heart_exclamation:

Our end goal is to help with all the above, and make sure more builders can easily join this ecosystem. We have been working on a proposal, and we will be opening a discussion on the Polkadot Treasury by this week, so we are looking forward to openly discussing it with everyone!

1 Like
10

@lucasvo, thank you for taking the initiative. We completely agree with your intentions, and they align well with the vision of DecentralWiz and DIVE that we are building at HugoByte. I would like to give a special mention to @alexdimes, who has been very supportive and is leading the initiative to enhance the developer experience.

As you have pointed out, as the core protocol gets enhanced, it is crucial that the infrastructure supporting the complementary services also be scaled to handle the increased bandwidth. Especially with dynamic core allocation, the DApps will still require a host of off-chain services for their functioning.

DecentralWiz, as an engineering platform and decentralized protocol, is precisely intended to address the problem you have mentioned. Along with easing the developer experience, DecentralWiz is committed to delivering solutions in a decentralized manner to circumvent various challenges such as the need to trust service providers, reduce deployment costs, lower technical entry barriers, overcome resource limitations, and provide easy access to utilities. DecentralWiz is to support a multi-provider ecosystem where the providers operate an open-source, zero-trust platform which is aggregated at the protocol level providing users a unified interface to deploy the environment which are isolated and secured even from the providers specifically tailored to run mission-critical and financially sensitive operations such as running collator nodes, validator nodes etc.

DIVE, as a component of DecentralWiz and with an integrated standalone CLI, aims to enhance the developer experience right from the early stages of development. Upon complete implementation, DecentralWiz will enable developers to effortlessly deploy environments in the cloud, hosting all the required services such as nodes, explorers, indexers, etc.

We hope you and the community will support our contributions. I am adding links to some of our work below that aligns with this vision. We look forward to continued support.

Decentralized Futures Discussion: Decentralized Futures: Chain & Platform agnostic Web 3 Engineering Platform
DecentralWiz POC: PoCs/polkadot-ocw-poc/substrate-node-template at master · HugoByte/PoCs · GitHub
DIVE CLI: GitHub - HugoByte/DIVE: DIVE deeply into the world of Blockchain and Web 3.0 using Deployable Infrastructure for Virtually Effortless blockchain integration
Deployment Template Package: GitHub - HugoByte/polkadot-kurtosis-package: Polkadot Kurtosis Package
Event Indexer and Workflow Orchestration system: GitHub - HugoByte/aurras: Aurras system

2 Likes
11

Thanks everyone for the feedback so far. When looking at hurdles for adoption in Polkadot and easily launching apps, I would loosely categorize the challenges in the following category:

  1. Collator bootstrapping
  2. Full Node Infrastructure, maintenance and monitoring
  3. Governance and other tooling
  4. Block explorers
  5. Indexers/Offchain infrastructure
  6. Wallet support, Custodians
  7. Templates to launch a chain
  8. Exchange support, On/off ramps
  9. Developer Experience, Testing frameworks and other developer tooling
  10. Financial support (subsidizing parachain launches for example)

What else are we missing? I’d suggest once we have these loose categories we can try to map out where problems are already being actively worked on and where we need push for new initiatives.

2 Likes
12

Hey @lucasvo,

I noticed that you listed 10 key components for supporting the Polkadot ecosystem:

  1. Collator bootstrapping
  2. Full Node Infrastructure, maintenance, and monitoring
  3. Governance and other tooling
  4. Block explorers
  5. Indexers/Offchain infrastructure
  6. Wallet support, Custodians
  7. Templates to launch a chain
  8. Exchange support, On/off ramps
  9. Developer Experience, Testing frameworks, and other developer tooling
  10. Financial support (subsidizing parachain launches, for example)

I wanted to bring to your attention that @MuhammedIrfan and his team at HugoByte are actively working on building and providing solutions for all of these components. I think you might have missed his post and the projects they have completed.

If you have a moment, I highly recommend checking out his post here and exploring the work and offerings of HugoByte. It would be great to hear your feedback and comments on their initiatives.

Their contributions seem to align closely with the key areas you identified, and I believe their efforts could significantly support the growth and development of the Polkadot ecosystem.

1 Like