Moving FRAME out of Substrate

Ecosystem
1

Not my idea, I stumbled across this discussion by @xlc and I’m here basically to re-post it so it doesn’t go stale, I’m also pretty sure it’s been discussed in other channels already and offline but here it might be a better place to talk about it.

:point_down:Original post by @xlc

Currently FRAME offers a set of generalized & reusable pallets but the size is still small and we don’t really have much external contributors submitting pallets to FRAME. A few reasons that I can think of are:

  • Making PR and getting it merged to Substrate can be hard. Core devs are busy and don’t always have time to review the change.
  • We don’t have any predictable release cycle for Substrate. This creates a big uncertainty on when the newly contributed changes will be available to downstream projects.
  • Pallets in Substrate should be generalized, but that come with cost of making code harder to write.
  • The pallets in Substrate cannot use anything from Polkadot or Cumulus. i.e. No XCM related pallet.

To workaround some of the above issues, we at Acala created ORML aims to provide a set of reusable pallets that covers more use cases such as multi token and XCM utility. It is currently used by at least 7 live parachains we are start receiving community contributed pallets into ORML. However, this is still not ideal.

Problems (and rants)

ORML as a separate repo, it can be hard for external team to include it. In the end, it is another 3rd party deps that needs to be carefully managed to ensure the version aligns. Specially that Substrate is not published to crates.io and therefore people cannot use cargo to manage the versions. They need to manually pick the right branch or there will be compiler error.

While I believe most of core devs for Substrate are aware of ORML, the development are completely separated and we end up with a lot of similar but incompatible pallets. A few examples are:

And we need to remove features from ORML because they have been implemented in Substrate. Occasionally that’s due to our own contribution as we still want the features to be maintained in Substrate as much as possible.

On top of that, there are a lot of almost required pallet for parachains that are not provided by Substrate/Polkadot/Cumulus and end up been multiple incompatible implementations between all the parachains. I only checked a handful of parachains so this is not a complete list.

And we have different flavor of collector-selection, xcm utility etc. Everyone are referencing each others and making their own pallets but with their own twist to fit their own requirements. This means wallets, blockchain explorers, any multi chain tools are very hard to be compatible with all the parachains. We have PSPs but I would leave it out for now as it isn’t really helping.

All the Substrate developers needs to unite together to change the current situation so that we can focus on our own core business logic, instead spend all the time to implement logics required to run a L1 such as token handling, governance, XCM, etc.

I believe most of the Substrate developers are aware most the issues I raised here, but the fact we are still here is mainly due the the issues are not easy to solve. But the issues aren’t going to solve itself so I am making this a discussion post, trying to identify where we want to be, and laying down a path towards there. Please help contribute your ideas to refine the tasks and help contribute followup issues and PRs to make this happen. But of course we will also need full support from Parity team as it requires restructure of the Substrate repo as well. Please signaling your support by upvote issue discussion.

1 Like
2

To start somewhere, one apparent small detail of restructuring Substrate and taking out functionality to different repositories is where are those repositories going to live? With the fellowship around the corner and having its own Github org(that I suppose should host more than just the manifesto), would it make sense to have FRAME be moved there? either as a mono-repo(e.g. https://github.com/polkadot-fellows/frame) or set of repos grouped in some way?
I guess its easier said than done, so perhaps we can start listing what the implications of such a migration would be?

3

I’m not speaking on behalf of Parity when I say this, these are my personal opinions.

It seems to me that the two conversations of whether to split and where an independent FRAME would live are orthogonal.

The fellowship isn’t a legal organization and at the end of the day Substrate and FRAME are Parity products and IP, so I think it’d have to be a really strong case for Parity not to own these. But it seems reasonable to me for Parity to commit to give merge rights on Polkadot-related repositories on the basis of fellowship membership - perhaps to all III Dan or higher. Again, just my opinion.

As for whether to split them, I think at this stage of maturity the case for that is becoming much more clear. It should not be very often that host functions in Substrate change, which require frame-system and pallet updates.

4

Thanks for reposting this @olanod.
Here is another related document I created Pallet repo re-structure - HackMD

Also I think it makes perfect sense for fellowship members to have some maintenance permission over the Polkadot & related repos. It will be great if the permission is determine by the rank. e.g. II Dan have permission to add labels, III Dan have approve PR permission.

This should doable with some custom Github Apps & bots. Parity have already built something similar which is a solid foundation for this.

1 Like
5

Fair enough, although I’d say they’re rather acute and related perhaps I’m getting ahead of myself and it’s a topic that can be revisited later.

But still out of curiosity, does Parity or any legal entity need to own the code? the concept of ownership in the open source world is a bit fuzzy and underrated IMO. Parity can own the repositories and the fellowship keeps forks of not only Substrate and FRAME but all other relevant repositories, it might be that over time the fellowship grows in size while Parity shrinks and in an organic way the fork becomes the “main repository”. Also nothing stops high Dans with the enough permissions within the Github org to skip Parity and create releases of the forks and propose runtime upgrades to the relay-chains or system chains directly for example.

By the way, linking the Dan system with permissions in Github makes sense, it can be used to control who can propose new pallets, trigger automations, review changes, manage project/kanvan boards, create releases by merging changes from less privileged branches to more privileged ones, etc.

It’s great to hear that Substrate is mature enough to be able to consider this kind of split without causing an uproar, what could be the smallest first steps that Parity could take in this direction?

6

Just to answer on this specific point: this really doesn’t matter, as the client ↔ host functions interface must remain backwards compatible no matter what.

The only advantage in having the client and runtime together is that you can add host functions and make the runtime use them in the same PR, instead of having to open a separate PR for the host functions first, then merge it, then update dependencies, then update another PR.

While it’s an advantage during fast-paced development, this also encourages poor design. Instead of thinking of host functions as an interface that must make sense independently, this makes you think of host functions as a solution for your specific problem which you’re focused on, which is not a good thing.

In other words, updating host functions and the runtime independently forces you to “sleep on it”. And I think that many warts in our design could have been avoided if people had been forced to sleep on it.