We would like to announce that we’re currently working on merging the Polkadot stack repositories (Cumulus, Polkadot and Substrate) on GitHub into one repository.
Why?
Merging the interdependent repositories into a single repository will simplify the development process, eliminate the need for “companion” PRs across multiple repositories, and improve collaboration among team members. A single repository will also make it easier to manage issues, pull requests, and documentation.
When?
We can’t tell this yet, we just started off with the work and there’s a lot of open questions yet to be answered. We just wanted to inform you that we initiated this project and we will keep you posted as we progress with the work. Once we have a functional (as we call it) mono repo, we will also give you guidance on how to migrate to our new stack
Updates will be posted in this thread. Feel free to ask here if you have any questions.
I am really confused about the general direction with all the repo structure & management. Have you tried the proper solution? i.e. Proper release planing, decouple the code, avoid unnecessary breaking change, etc?
Yes this is still part of the plan, and was perhaps not specified in the original post: runtimes should and will be controlled by the fellowship. I suppose this will be done after the mono-repo.
IMO It is hard to address that as it translates to doing a full engine repair while a car is moving. There is a tradeoff between development and stability, and many still expect new features to be shipped and we have to make a choice between the two. For sure we are closer to being at a stable position now, but we are certainly not there you.
That being said, this needs to be assessed at various scales. Making each crate fully independent, and semver compatible with release planning is something beyond any of us have the resources to do right now. I am trying to propose something like that just for the common FRAME APIs here: new doc/api crate: `frame` by kianenigma · Pull Request #14137 · paritytech/substrate · GitHub.
All in all though, I think the matter of releasing and versioning is somewhat decoupled in my mind. We can break the repo into smaller pieces, and still force them to stay in sync with companions and never have proper releases. And similarly, moving to a mono-repo does not necessarily mean we are moving further away from releasing. For further reference, see here about releasing: Publish Substrate to crates.io
We also have to face the reality and that is that we are still delivering quite a lot of features. However, I already have seen that we don’t break things that often anymore. However, given that we use Rust there are also some things like the Config trait that make it technically hard to not have a breaking release by adding there a new config option. However, there is also work being done and maybe we can improve on this in the future. We are also working on all “the fronts” to improve the usage of our stuff or at least to make breaking changes easier to integrate.
I also think this is a move in the wrong direction. Fair minded people can disagree on which is appropriate, but the perspective you adopt influences how you view this.
This move, it seems to me:
is natural if you take the pov that Substrate is the Polkadot SDK. Sometimes expressed as Polkadot=HTTP/S, “on-board the world to Polkadot”, etc. Here you might even say that to argue the relaychain is the unit of development is a fundamental misunderstanding. That is all fine, from this Polkadot-SDK perspective. Here Polkadot is primary and Substrate subordinate.
is unnatural if you take the pov that Polkadot is a poc of what you can do with Substrate. Here you would think of Polkadot as a website built on nginx modules rather than HTTP/S. Here you do envision multitudes of non-Polkadot connected relaychains. You don’t care if they run using public validators or private, interconnected or not, if they have public coins or not, if they are publicized or you never know they are being used. That is all fine, from this Polkadot-PoC perspective. Here, Substrate is primary and Polkadot is, at best, first among equals by virtue of being the PoC.
These two perspectives influence the tradeoffs you prioritize as you address problems.
The problems described are real.
The proposed solution only makes sense if the envisioned end state is Substrate as the Polkadot SDK.
I fail to see how this merge will make it easier to collaborate between non-Polkadot-centric networks.
To be fair to the proposal, it does not cite that as an objective.
If Parity, W3F and Polkadot devs are finding it difficult under the current code & repo structure then any other non-Polkadot-centric team will also find it difficult to contribute their ideas/needs.
So a solution is needed.
That solution will require different tradeoffs. How you weigh those tradeoffs will depend on the perspective or end-state you envision. Example, accepting some friction and loss of velocity in order to remain open to non-Polkadot-centric networks (admittedly this is a chicken-or-the-egg first problem - hence the importance of vision/perspective)
