We would like to announce that we’re currently working on merging the Polkadot stack repositories (Cumulus, Polkadot and Substrate) on GitHub into one repository.
Why?
Merging the interdependent repositories into a single repository will simplify the development process, eliminate the need for “companion” PRs across multiple repositories, and improve collaboration among team members. A single repository will also make it easier to manage issues, pull requests, and documentation.
When?
We can’t tell this yet, we just started off with the work and there’s a lot of open questions yet to be answered. We just wanted to inform you that we initiated this project and we will keep you posted as we progress with the work. Once we have a functional (as we call it) mono repo, we will also give you guidance on how to migrate to our new stack
Updates will be posted in this thread. Feel free to ask here if you have any questions.
I am really confused about the general direction with all the repo structure & management. Have you tried the proper solution? i.e. Proper release planing, decouple the code, avoid unnecessary breaking change, etc?
Yes this is still part of the plan, and was perhaps not specified in the original post: runtimes should and will be controlled by the fellowship. I suppose this will be done after the mono-repo.
IMO It is hard to address that as it translates to doing a full engine repair while a car is moving. There is a tradeoff between development and stability, and many still expect new features to be shipped and we have to make a choice between the two. For sure we are closer to being at a stable position now, but we are certainly not there you.
That being said, this needs to be assessed at various scales. Making each crate fully independent, and semver compatible with release planning is something beyond any of us have the resources to do right now. I am trying to propose something like that just for the common FRAME APIs here: https://github.com/paritytech/substrate/pull/14137.
All in all though, I think the matter of releasing and versioning is somewhat decoupled in my mind. We can break the repo into smaller pieces, and still force them to stay in sync with companions and never have proper releases. And similarly, moving to a mono-repo does not necessarily mean we are moving further away from releasing. For further reference, see here about releasing: Publish Substrate to crates.io
We also have to face the reality and that is that we are still delivering quite a lot of features. However, I already have seen that we don’t break things that often anymore. However, given that we use Rust there are also some things like the Config trait that make it technically hard to not have a breaking release by adding there a new config option. However, there is also work being done and maybe we can improve on this in the future. We are also working on all “the fronts” to improve the usage of our stuff or at least to make breaking changes easier to integrate.
I also think this is a move in the wrong direction. Fair minded people can disagree on which is appropriate, but the perspective you adopt influences how you view this.
This move, it seems to me:
is natural if you take the pov that Substrate is the Polkadot SDK. Sometimes expressed as Polkadot=HTTP/S, “on-board the world to Polkadot”, etc. Here you might even say that to argue the relaychain is the unit of development is a fundamental misunderstanding. That is all fine, from this Polkadot-SDK perspective. Here Polkadot is primary and Substrate subordinate.
is unnatural if you take the pov that Polkadot is a poc of what you can do with Substrate. Here you would think of Polkadot as a website built on nginx modules rather than HTTP/S. Here you do envision multitudes of non-Polkadot connected relaychains. You don’t care if they run using public validators or private, interconnected or not, if they have public coins or not, if they are publicized or you never know they are being used. That is all fine, from this Polkadot-PoC perspective. Here, Substrate is primary and Polkadot is, at best, first among equals by virtue of being the PoC.
These two perspectives influence the tradeoffs you prioritize as you address problems.
The problems described are real.
The proposed solution only makes sense if the envisioned end state is Substrate as the Polkadot SDK.
I fail to see how this merge will make it easier to collaborate between non-Polkadot-centric networks.
To be fair to the proposal, it does not cite that as an objective.
If Parity, W3F and Polkadot devs are finding it difficult under the current code & repo structure then any other non-Polkadot-centric team will also find it difficult to contribute their ideas/needs.
So a solution is needed.
That solution will require different tradeoffs. How you weigh those tradeoffs will depend on the perspective or end-state you envision. Example, accepting some friction and loss of velocity in order to remain open to non-Polkadot-centric networks (admittedly this is a chicken-or-the-egg first problem - hence the importance of vision/perspective)
IIUC your main concern is that the monorepo “Polkadot-SDK” would repel non-Polkadot devs that just want to build on Substrate?
Yes that could happen… I dont see an easy way to prevent it either; it is an unintended side effect of moving everything into one repo. Although I think the benefits in this case from easier external contribution and development workflow justify it.
The current repo structure with the companion merge requests are a pain to develop against. There are more than enough examples of it that prompted a change.
I am not really in the position to clarify to long term relation between Substrate and Polkadot, so going to tag @gavofyork.
I would invite you to challenge these assumptions. In my (limited) experience they originate from a team culture that is fast-moving and then struggles to make the hard decisions to bring more structure into their processes.
I have had a few conversations with Substrate engineers from different teams and many expressed some degree of unhappiness with the release process.
In the past, I’ve been part of a team where we tried to transition to a more structured process and failed a few times. But eventually, our management had enough and decided to introduce stricter processes for good. It required a good deal of organizational development, but we went out of the process far stronger, with a clearer structure and less stress for everyone involved, because we knew what we could rely on in terms of timelines/roadmaps/feature planning.
How this looks is different for every org, but I feel like arriving at more reliable release management is in order soon.
I was very much a proponent of giving Substrate its own name, repo, branding and project back in 2018. I absolutely believe there were (and are) good reasons for this. However we must be pragmatic and the reality is, after long consideration of moving to a monorepo (over the course of years), for various reasons (not least the painful and tedious companion system) we have concluded it’s the right plan.
I do not believe altering the name of the repo and combining with Polkadot and Cumulus will materially alter the likelihood of serious contributions (and might actually increase it given that it makes breaking changes much easier to merge).
Substrate will remain its own project, even if it doesn’t have an independent repo. We do plan on making proper releases of crates to crates.io, facilitating the use of Substrate among both parachains and solochains alike, and it is at this level where I think there’s a reasonable argument to present Substrate apart from Polkadot.
As for the repo name, “Parity Polkadot SDK” seems to me the best name for describing the code which is contained in it (basically, that’s Substrate, Polkadot, Cumulus and XCM). I don’t think it’s unreasonable to name the overall product after the project which funded the development of the code.
I generally agree with your comment that our direction should be toward more stability and I am personally not against it. If anything, my personal preference is highly in favor of more organized development, but I am trying to asses what would benefit the entire ecosystem and Polkadot the most.
As a small step toward this direction, please see:
To those who might perceive this as a harbinger of increased instability, I would like to respectfully disagree. The primary objective of this initiative is to alleviate friction within our development process. This friction, I believe, serves as a major impediment in achieving the enhancements that we genuinely seek: refinement in versioning, improved release strategies, releasing to crates.io, and the potential introduction of an umbrella crate.
These improvements would likely lead to a looser coupling of components, consequently reducing the likelihood of breakages for downstream crates. This decreased coupling might even offer us the flexibility to independently migrate components to their own repository, should we decide to do so. However, unlike previous instances, we could now avoid enforcing synchronization through CI.
As a solochain developer, my exact concern was Parity dropping its commitment to offering Substrate independently of Polkadot.
Seems to resolve that. Things will get worse from my UX perspective, as I maintain a fork of substrate and will now have to technically maintain a fork of an entire stack I don’t work with (though it should be simple enough to adapt to).
We will transition to the new repository (which will be named Polkadot SDK) on Friday, Aug 25th. Latest by Monday, Aug 28th everything should be set and ready for you to migrate as well. For the meanwhile we’ve created this FAQ which will be updated and adapted as new questions come in.
Issues are still accessible even after archiving the repo. So issues that are already assigned to a project board might remain in the old repo, other relevant issues will be migrated to the new repo by us.
I really don’t like it.
IMO maybe merging the polkadot and culumus repo is ok, but merging the substrate repo together doesn’t convince me, at least not with the comments above.
On the contrary, I think this will make paritytech contributors feel less bounded, which may lead to more breaking changes in substrate to affect developers in the ecosystem.
Maybe it’s too late to express my opinion, but I hope the committers of paritytech can consider merging polkadot and cumulus repo first.
I think he was primarily alluding to crates-io with this statement.
In the beginning maybe yes. But as soon as we commission the dedicated runtimes repo, it would lead to a cleaner separation since then semver is enforced.
The Polkadot runtime would have a normal crates-io dependency to upstream Substrate/Polkadot/Cumulus. Parity devs would thereby have 1:1 the same experience as ecosystem developers when it comes to breaking changes.
This should improve the situation IMHO by paying more attention to not breaking stuff.
