Underestimated developer cost in Polkadot ecosystem

Governance
1

@alice_und_bob has raised a very good discussion here:

However, as a parachain team, I’d like to add the other side of the coin: Not only the developer experience, but also the cost matters. I didn’t see any significant discussion in the Polkadot forum, so I’d like to open a new thread to talk about it. Actually I have shared my idea with Parity before as a follow up for the developer experience survey, but never get a response. So let me publish it.

I’m building on Phala Network and would like to point out some real problems not addressed in DX survey.

First of all, the real cost to develop on Polkadot is unmatched high. Compared with developing on Ethereum, parachain developers need to pay for the blockchain explorer, indexer, hardware wallet, and RPC providers monthly. Recently the OpenGov has rejected most of the initiatives to cover those costs by the relay chain treasury. So the parachain team has to pay. Otherwise each team must maintain a large DevOps and RD team to maintain those infrastructure. And don’t forget, running a network is already costly.

Secondly, the lack of standard results in very limited composability and consequently the ecosystem becomes further and further away from the “mainstream” users.

Although Polkadot is advertised as the interoperability solution, its deliverables are actually slower than the competitors. There are a bunch of reasons: the slow development progress of XCMP, the high barrier to open HRMP channel (governance is expensive), and the lack of XCM documentation, and the complexity of the three versions of XCM (compared with competitors like Axelar). We can name more reasons, but as a result, the XCM ecosystem evolves too slowly.

Currently the only way to compose two projects (parachains) is to use XCM. Given it’s hard to learn, develop, and maintain, it’s natural that the parachains evolved their ecosystem so slowly.

And let’s see how does the “mainstream” ecosystem performs. We know that’s the EVM ecosystem. In there, the projects are smart contracts, and they can interact with each other seamlessly. Contract calls are way easier than XCM and it doesn’t require any governance process. On the other hand, in the parachain ecosystem, a new team can only build their own parachain and compose with another by XCM, or they contribute to the parachain codebase by writing a pallet. The both process involve governance and thus a lot of resources to establish the trust in the community.

Each reason adds the cost to the parachain startup, and eventually the ROI of building a parachain becomes a questionable value.

I see the Polkadot 2.0 is a start point trying to address part of the problem I’ve pointed out, but at this stage, unless we take very straightforward measures, I’m not optimistic about the future, as a parachain. My suggestion:

  • Shift the ecosystem focus from pallet development to smart contract development. This removes most of the costs: auctions, infra, experience to learn XCM, etc. this means we shouldn’t all in XCM as the only way to build composable projects.
  • Strong support of the infrastructure like blockchain explorers, indexers, RPC, etc. It should be funded by the relay chain treasury if we want be competitive to attract any innovative developers.
  • Focus on standardization to further encourage composability between the projects. By standard, we should not only think from the Polkadot’s view, but also the “mainstream” users’ view. In other words, deploy the practical interoperability solution to the EVM world as soon as possible.

I really want to see the success of Polkadot. Hope my suggestion can help.

16 Likes
2

You are very right on the cost topic - Parity has promoted parachain building as their primary goal, and slowly started to move on to dApps lately. I had spoken to dozens of projects who wanted to become a parachain, and had to convince them that the cost of running all infrastructure that parachain are is in the millions of USD per year.
“Shift the ecosystem focus from pallet development to smart contract development” - that statement assumes there is someone to do that shift. As a matter of fact that function is now completely decentralized and considering there is no entity with resources to push for more parachains, my assumption is that the shift will happen organically. Parachains offer various options for building dApps, but the Smart contract approach is the most common one.
The second question you mention - standardization, is different. Indeed we have a zoo of tools people use, and most of them suffer for it since no one solution has the big investment and customer base to succeed, so we end up having 10 different tools for the same thing that are all poor or worse. On the topic of common infrastructure we as parachains should come together and come up with the strategy and the plan what should be supported by the Treasury, and convince our communities to vote to support that.

2 Likes
3

Good points and I agree in general. A few comments.

I think it is possible and should be a suggested standard solution. If you don’t know if you should build smart contracts or a chain, you SHOULD build smart contracts, because you haven’t understood your problem well enough yet and therefore should walk the simpler path.

The key driver for this can be smart contract parachains, who already have some ecosystem fleshed out and can be ones who create XCM connections to other chains.

I disagree with this point, as it would remove the pain that is necessary to drive innovation here. Projects like Tanssi aim to remove some of the pain here (shared sequencers would also be a solution), and if the relay chain subsidizes infrastructure, it would remove the burden of innovation.

Yes, there should be more standardized procedures around how tokens are represented and how channel openings and tokens can be registered on another chain. MultiLocations are a first step, but having better conventions around the default MultiLocation of a token would help a lot in reducing overhead and developing standardized interop.

It’s hundreds of thousands, but not millions.

1 Like
4

I can speak to the costs of running a parachain in the Polkadot ecosystem, we are practically broke at this point after embarking on this course 4 years ago - hence our Hail Mary proposal.

Successful or not, that proposal should be a wake-up call, something needs to happen. Rather than talking about it, I would like to see other teams proposing solutions as we have. We took a very big risk in exposing our problems, but I know we are not alone, and this thread hints at this.

As I said in the polkassembly comments I am extremely sad at being in this position after so much time effort and money has been poured into our parachain and the thought of shuttering it (in favour of solochain, or whatever) is not nice.

2 Likes
5

Easy math:

  • $250k to get a decent infra setup: RPC nodes, explorer, governance frontend, indexers.
  • Crowdloan interests: $400k (info)
  • R&D: Depends on location, but you can consider a team of 10 dev is typical. If just count the parachain overhead, 2 substrate dev, 1 frontend and 1 devops, all dedicated to parachain maintenance. If each get $150k/yr, then that’s $600k/yr.
  • Not counting wallet and hardware wallet integration fee, which is basically zero on EVM chains.

If run on both Polkadot and Kusama, some costs would double.

1 Like
6

if you add R&D to infra costs, it checks out. I thought you were talking about pure infra.

7

In a broad sense, I agree with the some of the thesis share in this post.

  • Development Experience is suboptimal, Development cost is high.
  • More emphasis on smart contract is a promising way forward.
  • More standardization helps.

However, I want to push back on the type of thinking that is being used to lead to these conclusions.

Compared with developing on Ethereum, parachain developers need to pay for the blockchain explorer, indexer, hardware wallet, and RPC providers monthly

Comparing the development and maintenance cost of a parachain to a smart contract is similar to comparing the development cost of a bicycle and a car. Of course, one is more expensive than the other one, but one can also do many more things.

I don’t deny that better standardization and tooling can improve the parachain development cost. A good recent example of this is to finally have a proper multi-parachian ledger app (see Polkadot Generic Ledger App). But, I highly doubt if it will ever be as simple as deploying your application as a smart contract.

A similar conclusion can be drawn about XCM as well:

Although Polkadot is advertised as the interoperability solution, its deliverables are actually slower than the competitors

I don’t want to go into too much detail, but I am sure it is reasonable to acknowledge that the tools and measures needed to let two blockchains talk to each other is significantly more complicated than two contracts in the same blockchain talk.

I think these complications about the true cost of running a parachain were discovered somewhat late, and more importantly, not educated in our ecosystem fast enough. Nonetheless, it is better late than never.

So, the problem is really not JUST that parachains are too complicated. Part of the problem is also that the use cases that were fitted to be a parachains were too simple and therefor not worthwhile the cost of running in.

What I personally hope to see is a first-principles debate around this topic, one tries to find the right fitting for “what application should actually be single parachain?”.

That is all not to say that we should not try and improve the situation, or try and help the existing teams survive and re-evaluate. We should, but we should also look back and objectively look at what we have done wrong in order to improve.

I think @alice_und_bob pointed out the underlying crux of all of this fairly well:

If you don’t know if you should build smart contracts or a chain, you SHOULD build smart contracts, because you haven’t understood your problem well enough yet and therefore should walk the simpler path.

Teams should be encouraged by the ecosystem to prefer building a contract (be it ink! or Solidity) if they are not sure how much scaling they want, or how much funding they have. Hopefully in the next year, with on-demand/agile-coretime a team can have a smooth transition from building with ink! → deploying on a parachain → partially migrate to a hybrid chain that is their contract + some pallets, and be on on-demand parachain. Finally, if needed, they can upgrade to a bulk parachain.

Taking the analogy above, if you are not sure how far you want to go, you should choose the bicycle and not the car.

7 Likes
8

In my experience launching a parachain, the costs around the development is not a major issue.

I was lead engineer (midwife) of the kaobcha parachain on kusama, which began as an experiment to see if you can do it without all the funding and hype. And the answer is yes and no.

Yes, you can build, test and launch it with 1 to 3 technicians, depending on the initial complexity of the parachain. All in all probably cost around $50k from grants on the technical side. Theoretically the only funds you need is to: post the deposit for a PARA_ID, launch parathread, enter slot auction/crowdloan, which is like 200KSM ($5k), in fees and deposit. Then you can win an auction for almost nothing depending on demand. the second kabocha auction was won with just 1 KSM.

However, the issue is then having funds to add to liquidity pools, buying DOT or KSM to fund the sovereign account; funds to pay the centralised exchanges big amount of money to go on the exchanges. So really, the cost problem is not really a technology one, its the cost of kickstarting the social money game.

Another $200k to kickstart the token would have made it more interesting an experiment.

The parachain was launched on a shoe string budget with a grant from another chain called Edgeware, and cost a few servers to run on Digital Ocean droplets, An RPC, 2 x Collators, and an RPC Archive node, all in all probably around $300 a month.

Block explorer can be free, you can just make a PR to polkadot JS, or you can fork it and run your own custom version on a server, but that does not cost very much.

the parachain is still ticking along but without a funded token.

If the kusama/polkadot treasury could fund bootstratpping the liquidity pool then theoretically you dont need any money to launch a parachain, as long as all the initial team is willing to work for future equity. If the launch is a success and the token gains traction then the the treasury becomes available. Then the problem is demand for the token versus the sell pressure for funding work.

Imo, its valuable for the main treasuries to fund some of these grassroots projects, because these projects have already provided a signal that they worked on something without being paid, meaning that there is a decent chance that they will continue to work in dry spells, and that they are intrinsically driven. So some kind of long term expectation value should be applied. Also the learning curve of these people is continuing to grow, because launching a parachain requires a certain desire to grow and put yourself in position you’ve never been in. I would back horses like that.

Personally, i moved on to work on a new project, but the kabocha is a good example of how you can experiment in this space on a grassroots level.

7 Likes
9

You should not forget the cost of staff. Also whether you like it or not, as soon as you publish your RPC endpoint all the wallet providers use it in their apps. This means you have to have proper web style front end load balancing, monitoring and all the rest just to ensure that those nodes are not taken offline by Jaco’s insane “remove an endpoint if it’s overloaded and I cannot reach it” script.

Also you still haven’t accounted for a decent quality and development environment so all those costs are 2x-3x.

10

This is very unrealistic. There is no way that all nodes you propose cost $300 per month in total. Each one maybe at a push but if you undersize your node you will find it falling over and running out of space very frequently - and that’s something that cannot happen to boot nodes, RPC nodes or collators. So it costs far more than this for a stable / professional setup.

11

There are many VPS options offer instance with 1TB storages with good CPU and RAM for less than $100 per month. With Subway, it is easy to scale the RPC nodes and it does the failover and dead cheap to run many instances. It wouldn’t cost more than $150 to host 2 RPC nodes and many Subway instances and have a very available and fast RPC service.

1 Like