Parity is currently operating Rococo and Westend as test networks. Westend was the first test network that was used as a testing ground for Polkadot/Kusama upgrades before they hit main net. Later, Parity introduced Rococo as a dedicated test network for parachains as the parachains consensus etc was quite unstable and it could may be restarted at any point. Both of these test networks are being operated mainly by Parity and it is open for everyone to join as a validator for example. Parity also has control over the Parachain slots on Rococo and who is on boarded there. Over the time it happened that Rococo and Westend stopped for some amount of time because some updates broke them. They served exactly the purpose they are being created for, for testing out new shiny stuff and if it breaks, it gets fixed. Parity doesn’t provide any guarantees on the uptime or similar. However, the community would like to have some test network that is “stable” and more conservative when it comes to updating, similar to Polkadot.
So, the community should come together and create a test network that is entirely controlled by the community. From the community, for the community. In the very early days of Rococo, I think around 2020, Rococo was also open to everyone to get a parachain slot, but the only requirement was to provide two validators that would be added to the set of the relay chain. So, everyone would have some stake in the game to ensure that the network would run smoothly. This test network could also be used for operators to try out running a validator etc.
This post here should mainly be used as some sort of discussion starter around this topic.
From Trace Labs (core developers of OriginTrail) perspective, we require a stable - “staging” if you will - relay chain environment that would be considered as close to productive as possible, in order to promote it’s usage with our partners and clients as a “staging” environment for their builders (and our own as well, as we are a builder too). Effectively this environment would be the same as Polkadot (prod) in both features and in perspective of maintaining it.
Perhaps a term “stage-net” if a bit better than “test-net”. Think → “sandbox for users” for which there is an easily accessible token faucet etc, which would make it also easy to use and at the same time be “reliable”.
Currently Rococo seems not to be this network and we’d really like to have one such relay chain, and asap. Happy to support by running validators etc.
I’d really like to hear what others think and if there are other teams with such needs.
@bkchr I would like to see a CoreJam/Coreplay only test network more or less as soon as possible. This would be of a completely different character than the parachain focussed ones, and not need to get bogged down with System chains, parachains, opening channels, or even XCM. I know it will take some time to develop this (pressure), but Data Availability is central to the CoreJam programming model, and I think splitting off this gem from Polkadot 1.0’s parachain needs would also allow us to develop a PoC of how DA can be used for non-Substrate stacks (OP Stack, StarkNet, etc.).
CoreJam promises to be a virtualized smart contract interpretation layer (the old Solidity/EVM + ink/WASM=>PolkaVM) but also Coreplay, and both are central to 2.0 future.
Its really important that this 2.0 future not get bogged down too much with its 1.0 past. We don’t want a situation where any CoreJam / Coreplay innovation is kept back because it could possibly affect some parachain’s need for robust testnet.
If you can give some of us guidance about how to jump into CoreJam/Coreplay development (and its underutilized DA layer) especially with the “Decentralizing Parity” tectonic plates movements, more than a few of us can probably execute on it.
Thank you for asking that this be a community effort!
OnFinality previously ran a private relaychain network that matches your requirements exactly back in 2021 for parachains before the original slot auctions were held. From memory I recall that there were 5 networks participating in it and costs were shared across different participants. We terminated it once the auctions were completed and slots assigned to mainnets.
If there is demand for this, we can certainly look into this again. Are there any other parachains (or prospective parachains) that are looking for a stable and dedicated staging network that they can share with third parties?
Agree on a staging network - we currently use Rococo for this purpose as a constant “test environment” for HydraDX. All of our public and internal testing is conducted on here in final preparation for our own runtime upgrades. (in addition to chopsticks tests ofc)
A stable environment that mirrors the current Polkadot runtime would be perfect!
So if I understand correctly, Rococo would continue to be a “testing environment” for relay chain upgrades, and Stronknet would become a relaychain mirror for parachains to host their own “testing environments”.
I am part of the team operating the Rococo, Westend and Wococo testnets at Parity.
I think such a “community owned and decentralized testnet” would be useful. The validator community has long been asking for a permissionless testnet that could serve as a staging environment. This would let them have a risk free realistic environment to test changes to their validator setup. I think our best bet to make this work would be to capitalize on their experience and have them lead this initiative.
However, for this to succeed, we need to first think carefully about the following:
Code stability: only code that has been fully validated on earlier unstable testnets (Rococo, Westend) should be deployed on this new “stable” testnet. This will require a proper process to be defined by Polkadot developers and stakeholders.
Agreement on the network parameters (eg. should we have faster epoch times than Kusama to be able to detect potential issues earlier?). Unfortunately some of those chain configuration parameters cannot be changed after launch (to my knowledge).
Incentives: operators running bootnodes, validators, collators, RPCs should be either guaranteed compensation or have self-interest in maintaining their nodes up and running (eg. keeping good reputation as a validator operator or using it as a staging environment)
Governance: who would own the Sudo key? We could imagine a Proxy to Sudo or a Multisig to Sudo (eg. through Multix) that would be entrusted to a limited set of people (maybe the fellowship or another on-chain collective members ?)
Monitoring: we should be collectively informed when the network is down or degraded in order to take quick action
Coordination: we need the communication channels to synchronise on upgrades and collectively fix issues when they arise.
Yes, stage-net sounds like the right story, so exactly the same pallets, etc. as polkadot, but what validators & what security parameters?
We’d have validators operated by all parachains presumably. We could try to have as many nodes as polkadot, but this maybe tricky. As the network is not value bearing like kusama, we could tune down needed_approvals so that more parachains can run on fewer validators. We do not need all parachains running all the time either I guess.
You guys are talking about Kusama without talking about Kusama to a degree that is frightening.
I mean… with instantaneous coretime around the corner it will become very simple to acquire some cheap blockspace for a limited amount of time to test out stuff.
The thing is Kusama = the “chaos” network.
We need a non-chaos network, something considered as a productive clone, that the attitude towards is as if it were a production environment. IMO this network would require someone taking responsibility to minimize chaos, as if it were prod. So far Kusama doesn’t seem to be that.
Two ways to do it:
either each team runs their own relaychain, taking responsibility on their own and making sure things are alright (probably easier as it requires less coordination, yet only 1 parachain connected means not exactly a “copy of production conditions”)
or a common relaychain ran by multiple teams, that would follow Polkadot 1:1 in terms of code and deployment (harder, but closer to Polkadot real env as it has multiple parachains). This would be a preferred option in our view, yet is a matter of dedicating resources from multiple teams towards this
I think the difference between the proposal here and Kusama is primarily having or not having a value bearing token.
It seems that we would want a stable stagenet which has no inherit value on it. Is that right?
Otherwise yes, I think Kusama existence is exactly to provide some sense of technical guarantee to Polkadot, and I am still surprised at the chains which obtain a slot on Polkadot and not on Kusama.
This is indeed a very interesting discussion. We are currently building Portico, a protocol that has the intend to help Polkadot become as easy as possible to use for builders. We are a team with vast experience in Polkadot and Substrate (including with the inner-workings of Rococo!), and deeply believe that for Polkadot to be successful, developer experience is key. A stage-net as proposed here can help the ecosystem get there faster, and we are here to help.
Our objective is to build, run and coordinate a stage-net that has the exact same runtime as Polkadot, but with the add-on of the Sudo Pallet.
For establishing this community testnet, certain design decisions and requisite tooling are essential to ensure a successful network operation managed by the community. We were here when previous experiments were conducted as community networks on Polkadot, and we want to leverage the learnings from back then so as not to repeat the same errors again. We envisage the following requirements:
We need to strive a balance between having a stable stage-net and releasing the new Polkadot runtime sometime before it is deployed on Polkadot so that it can be pre-tested. Therefore we would like to discuss with the community, what is the best way of ensuring a new Polkadot Runtime is ready to be released on this stage net.
Should we upgrade right after the fellowship releases the new runtime? This would be the fastest option. Potentially there could be some dry-run with Chopsticks before launching on the stage-net.
Maybe after other test networks like Rococo or Westend have done their upgrades? Do bear in mind that Rococo does have some divergent features and therefore is not 100% compatible. Yet it gives more certainty on what is working on the proposed new runtime.
Or perhaps after Kusama but before Polkadot? This would be the slowest option, but the safest in terms of ensuring the runtime does not have any critical bugs.
Deployments as soon as a release is done on the fellowship. This will make it so that the stage-net will have the future Polkadot runtime ~2 weeks in advanced. No special features, just anything that Polkadot will ship with. Ideally the runtime upgrade might be tested before the enactment using Chopsticks and only proceed with it if successful.
OpenGov and other features will be enabled, thus allowing for anyone to test these, however the action queues (referendum, scheduler, etc) will be kept accordingly with the periodic use of Sudo. This will allow the testnet to be always useful and be resilient to, for example, governance white-hat attack like the one Avail’s testnet suffered in the hands of ChaosDao.
We are decided and capable to run a high number of relay-chain validators, however we also believe that the parachain teams that want to be connected for long periods of time, should also run 2 relay-chain validators to be connected to the network. This will ensure that the network runs smoothly and in a scalable way. However, this requires:
Commitment to always update the node the the latest release.
Tooling to have some central sharing of logs. We will work on this as part of the proposal.
Ability to raise to the occasion if any issue is to happen.
This part if key if we are to work all together as an ecosystem on this.
Active, open reporting of network status and metrics.
We are prepared to go ahead and help the ecosystem with this. However we’d like to hear first feedback from everyone interested in this stage-net to make sure we propose a useful and comprehensive plan to the broader community. Feel free to comment on the design decisions posted before, and add the ones that you feel are missing.
This was initially outlined as the last step in the community rollout process for Rococo becoming a Community Parachain Testbed:
With the latest discussions, the expectation seems to have changed from Rococo being a pre-production relay chain to a post-production relay chain to Polkadot and Kusama.
I agree with Shawn’s determination of the proposal which is the token should have no value - not low, but actually none. This allows for a much better “live” development experience where there is no value assigned to any token, and can be a playground for partially-formed ideas and riskier/dry-run deployments.
Another bonus of having a community-owned stagenet which bears no value means we could potentially have features such as auto-upgrading nodes, a public logging service, and other types of management to provide better debugging value for developers, which isn’t possible/desirable for a mainnet.
In doing so, though, we should make sure we understand fully what the point of each network is, and have adequate onboarding tooling around it as well as a clear developer path including convenient tooling (for example, concrete guides on development cycle from local → stagenet → kusama → polkadot). This would potentially imply that this stagenet would receive features more quickly than Kusama, so developers can develop against the latest features, but an open question is how to balance that with the expectation of stability?
Community-run testnets are a great initiative, and they enjoy success and traction in other ecosystems (see Goerli/Holešky). It may be of interest to read about the aims of different Ethereum testnets.
I think this would be a nice fit for a new System Collective (as outlined in RFC#12). This collective could have permissions relational to the amount of work that they contribute to it.
The stage-net would then need a bridge to Polkadot such that the sudo key can be controlled by that collective plus maybe a fallback option locally.
@Ben: There are currently 3 main scenarios -there are others that are too internal, like Versi, but are not thought of from an ecosystem perspective- happening at the moment with regards to testnets:
Rococo. As a PoA testnet it has been working a lot better since what it was two years ago, and now it runs quite well. Today is the most widely used testnet by parachains, and it’s being managed by Parity. Still, it has in it’s runtime special new features that will not make it to Polkadot in the following release, and that can potentially become an issue, like what happened with Async Backing last week - but yet again, that’s what testnets are for!
Westend. The original NPoS testnet that Parity had, it currently has 16 validators (15 Parity and one Kagome). With this amount of nodes, it can’t serve a wide range of parachains connected to it.
Parachain-ran internal testnets. These are used by parachains themselves for internal testing. Usually follow a stable runtime. This is very convenient for teams to get all the logs from the relay chain validators if anything happens, however it’s a difficult model to scale as an ecosystem testnet.
We agree with @branarakic before that we should pursue a stage-net. It would have Polkadot’s runtime with the add-on of the Sudo Pallet to keep things working smoothly.
Who is running the validators / or sets of the validators?
We think we are capable to mantain a good number of validators (~50) and then parachain teams that want to use this testnet long term, should provide 2 additional validator nodes. There are some open questions and more details around our idea before on how to do this, but the TL;DR is that the network needs a good base operation (that’s one of the things that chachacha didn’t have in the past), but also needs to scale. It will require tooling to be built too.
Who will be in control, or how will the sudo permissions get managed?
I think this would be a nice fit for a new System Collective
@OliverTY The collective is a fantastic idea, something that we would definitely get behind and help execute! It could also potentially be the Parachain Technical Fellowship. The interesting thing about this is that it would be deployed as a Substrate<>Substrate Bridge on AH on this stage-net, therefore not being a dependency for the Relay Chain’s runtime. Big question is how can we make it so that tokens don’t flow from one network to another.
For the beginning though, and in order to make the network stable and reliable, we suggest in our proposal to control the SUDO key as a multisig with a set of known people in the ecosystem that will have the best interest of it working.
@OliverTY The collective is a fantastic idea, something that we would definitely get behind and help execute! It could also potentially be the Parachain Technical Fellowship. The interesting thing about this is that it would be deployed as a Substrate<>Substrate Bridge on AH on this stage-net, therefore not being a dependency for the Relay Chain’s runtime. Big question is how can we make it so that tokens don’t flow from one network to another.
I think you wouldn’t need to use a bridge. After all, this will be a testnet and as such won’t have any security guarantees so bridging to it doesn’t make sense. Instead, we should just use the Polkadot collective members account list as an oracle for the signatories controlling the sudo multisig. The technical solution to achieve this on the testnet will probably need to be centralized but it’s not that important. However, on Polkadot this will be an actual entity that could have a treasury and agency to make further proposals.
Correct, when we proposed the bridging solution for the Parachain Technical Fellowship, we were envisioning the most descentralized solution that it came to our minds and we expressed that idea as something for the future.
However, we agree your comments and your proposal about the oracle setting the collective members signatories sounds very much interesting (even if it’s centralized).
Still, for an initial phase we believe that a multisig controlled by a set of known people, would be the best strategy until we can migrate to any proposal like the oracle described by you.
This would potentially imply that this stagenet would receive features more quickly than Kusama|
As we discussed 1:1 I believe this is an important point to clarify - from the discussion in this post, I believe the request from the individuals is a, Staging Environment that mirrors production (Polkadot).
Totally agree we need to ensure the deployment path and environments are stipulated and very clear.