As a high level summary for the current state of this (as far as I know), the main blocker here has been the multiple separate repos which compose the Substrate / Polkadot ecosystem.
There has been talk of turning these repos into a monorepo for at least a short while in order to unblock all these publishing problems, and potentially over time spinning them out again to appropriately separate repos.
There’s no manual intervention within that whole publishing pipeline, so you can count on automatically having new crate versions at most a few hours after a new commit appears on master. Only if some unexpected problem comes up we’ll need to manually intervene. On this note, automatic publishing has been enabled since last Friday (Dec 15) and the pipelines so far went well, save for one whose problem is already fixed. Overall the implementation has proven to be stable enough to the point where I can actually vouch for crates’ comsumption through crates.io.
One crucial detail is that currently any crate change is considered a breaking change. This means that minor and patch releases have to be performed manually because that use-case is not yet covered by the automation. As far as I’m aware there’s not even a plan for how to automate minor and patch releases, so having that level of sophistication is potentially a ways off.
No effort has gone into implementing publishing automation for Polkadot or Cumulus so far. That effort would turn into a roadmap item and/or ticket, the same way it was done for Substrate, but I’m not aware of any development on that front so far. For the lack of a plan I also can’t say when it’ll start. I’ll ping the people responsible for that and ask them to engage in this thread with more information later.
Good to see some progress here but the work is useless for parachain teams if Polkadot/Cumulus is not updated to allow us to include Substrate from crates.io & Cumulus/Polkadot without duplicated deps.
At the ask of @bkchr we’ll be moving to weekly publishing instead of per-commit. That change takes effect immediately.
This is still true, but I can provide a guess for when it could start: next in year, in January. That’s when the Release Engineering’s team lead (Mara) should be back from holidays and then we’ll be able to plan out how the crates.io publishing for Polkadot and Cumulus would work. I can draft a proposal in the meantime, but I’ll remain mostly inactive until January.
What does “based on” mean in this case, exactly? Does it mean that we should only trigger the publishing of Substrate crates before the releases of Polkadot? Or that the crates’ versions should somehow by influenced by the Polkadot version? Or something else?
However, although they’re synergistic, Polkadot and Substrate are not dependent on each other. Polkadot parachains can be built and maintained without ever touching Substrate (though alternative software options for doing so are currently limited) and chains built with Substrate do not need to be connected to Polkadot or Kusama. Substrate-based chains can exist as ‘solo-chains’ on an independent basis.
That being my understanding, I argue that Substrate crates’ versions shouldn’t anyhow be coupled to what’s happening in the Polkadot repository. In fact, this is how it works currently. For instance, a while ago, before the automated publishing pipeline even existed, version 7.0.0 of sp-std was published ad-hoc; “7.0.0” isn’t matching any Polkadot version - the PR even mentions that the publishing was motivated by subxt, not Polkadot. “7.0.0” also isn’t used uniformly for all crates in the repository since each has crate its own history and development lifecycle, e.g. some are newer or more/less mature than others.
I think the current versioning scheme for Substrate, where each crate has its own version and each crate version is managed separately, makes sense since crates have different histories and purposes. Not all crates are related to one other. I don’t see a reason to go for “one version for all” because that would incur bumping the versions of everything even if we want to actually only publish a new version of a single crate (example); the current strategy of only publishing new versions for crates which have changed is obviously more efficient than having to republish everything, every time, due to a sporadic version bump from a random crate.
You are mixing there the Polkadot project and the Parity Polkadot implementation. The Parity Polkadot implementation is build on Substrate and there is a strong connection. Currently there are also no real Substrate releases and everything we do is “Polkadot centric”. This means people are building their stuff on our Polkadot branches in Substrate/Polkadot/Cumulus.
I see the switch to the weekly crates release just as an intermediate step as releasing after every commit is too much. Releasing every week would also be too much, as long as it is done automatically. In a perfect world we would release only crate versions when a human decides that a certain crate requires a new release. Maybe always enforced on a Polkadot release. But that would be too complicated for now and we should do this step by step.
The next step after the weekly crates release is that we release the crates when we do a Polkadot release. This would mean when there is a Polkadot release, we release the Substrate crates, we set the Substrate crates in Polkadot as deps, we release the Polkadot crates and then we do the same game again for Cumulus. Then we would have all crates in a coordinated way on crates.io.
All substrate crates come from the same monorepo. We already discussed this I think and we agreed to just treat them as one blob for now. Let us sync all those versions. It makes anyones life much easier. We can just say substrate version xxx. Instead of going to hunt for a compatible set of crates. This “only bump what you need” is a nice fantasy. But it is just that: A fantasy.
cumulus, polkadot, substrate is already a monorepo. Any change in a dependency repo (polkadot, substrate) requires a developer to make the dependees (cumulus, polkadot) compatible. This is a monorepo with extra steps. If we would depend via crates.io and it would be the job of the dependees to update their code it would be a whole different story.
Do we do this in addition to the weekly crates.io releases or instead of?
Does this mean the repos will be permanently linked via crates.io? Or do we keep git dependencies on master?
We don’t release to crates.io per pr. In general I would like that we remove version numbers from master at all and then the tooling puts in the version numbers when it releases the things to crates.io.