Outstanding Questions About the Infrastructure Builders Program (IBP) Bounty

Ecosystem
1

After reviewing the Infrastructure Builders Program (IBP) Bounty #50 in more detail, several questions have arisen.

1 - Pricing model (AWS / Google vs actual providers)

According to the last top-up proposal, the nodes pricing appears to be based on AWS and Google Cloud benchmarks.

p1
p11013×574 69.6 KB

p2
p21048×646 47.7 KB

However, according to the IBP dashboard, in practice, some services are delivered by Hetzner or even by operators without a clearly identifiable entity.

ibp1
ibp11812×726 74.4 KB

ibp2
ibp21275×488 38.7 KB

Why is the base pricing calculated using premium cloud providers if the services are ultimately delivered by significantly cheaper operators? This approach seems problematic, as it enables certain providers to capture the price difference without clear justification.

2 - Large price discrepancies for the same service

Tender #3 spreadsheet of Public RPCs for Relay and System Chains (Bounty #31) shows offers from node providers to run a Polkadot Relay Chain node at an average cost of approximately $2,200 per month.

Why, then, is the IBP Bounty paying between $4,000 - $5,000 per month for what appears to be the same service?

g1
g11920×869 86.4 KB

Doesn’t this indicate that the IBP base pricing model may be inflated, as previously suggested, given that providers are demonstrably able to deliver the same service for significantly less, as shown by the tenders under Bounty #31?

3 - Parachain RPCs

The IBP Bounty, and ultimately the Polkadot Treasury, is currently funding RPC services for parachains such as Hydration, Moonbeam, Bifrost, Nexus, Ajuna, Unique, and Xcavate.

Why are RPC nodes funded for some parachains but not others? Who makes this determination? Shouldn’t this responsibility rest with the projects themselves and their token holders?

b1
b11491×576 66.7 KB

4 - System chain RPCs

Do we really need 7 or 8 RPC providers per system chain?

5 - Potential double payments (LuckyFriday)

LuckyFriday is currently being compensated by the IBP Bounty for 23 services, including:

  • Polkadot Relay
  • Polkadot Asset Hub
  • Polkadot Bridge Hub
  • Polkadot People

k1
k11546×1458 208 KB

However, these same services appear to already be funded under Public RPCs for Relay and System Chains (Bounty 31). How is this potential overlap being addressed?

Spreadsheet Bounty 31
LuckyFriday payment - Bounty 31

6 - Curator paid $3400 per month
Finally, there is also the matter of Coinstudio receiving $3,400 per month as a curator, without it being clear what specific work is being performed in exchange for this compensation.

Given the significant amount of funds involved, I would appreciate some clarification to better assess whether this bounty is being managed properly.

4

  1. Those are the ASNs of those making DNS queries in order to utilize the service. :joy:

  2. IaaS pricing is based on utilization and utilization is dynamic. Utilization is updated regularly here and base pricing is here

  3. This was defined in the last top-up on pokadot #1516

  4. Yes, it’s for latency optimization to increase quality of service globally. This was discussed extensively in KSM #35, on AAG, and in other places.

  5. Lucky friday provides RPC services from their location in Arizona I believe and their services for the public bounty in their location in washington d.c

  6. This was resolved in another thread

With such basic errors I have to assume that you’re rage bait trolling and not actually challenged in any way. With that said, good one :rofl:

5

Sorry, but I still don’t fully understand why there are DNS queries going through Hetzner. Which providers are these services actually hosted with? Can someone verify this?

It’s also still unclear to me how the potential double payments involving LuckyFriday across different bounties are being handled. Likewise, I don’t understand how the same Polkadot RPC service can be paid $5,000 under one bounty and $2,200 under another.

Perhaps another member of the community can provide more information on this.

Thank you.

6

Those are the USERS of the service, not the PROVIDERS. We would never record ip addresses of actual rpc users however DNS presents an opportunity for data collection from a 3rd party. The DNS server isn’t technically the RPC client because of the way DNS works. Since we’ve implemented this custom DNS system it allows us to use the DNS query as a proxy for the user to get a general sense of location. (Metadata-esque) This is useful to get a sense of global load and how that changes over time so we can track load from countries / regions and determine how best to update things in the future. It’s also used by the DNS system to globally manage load and members. It’s important to note that these are dns queries not rpc queries. Our RPC query handling is into the billions a day. These are DNS lookups that are done prior to connecting to the rpc server and a single DNS query could represent 10s, 100s, 1000s or 10s of 1000s or even millions of rpc queries.

I mean, on the dashboard there’s a map that gives you the exact latitude and longitude of the datacenter for each member. You can verify it by looking up that member’s corresponding ip address. Not sure what else you could want…

Lucky friday has multiple datacenter locations as do many other providers. Stake Plus has servers located in Washington DC and Cincinnati OH. Dwellir has locations in Sweden, 2 in Africa, and some others that I can’t recall right now. Lucy friday is providing IBP services from one location and providing the public bounty rpc services from another location. 2 sets of hardware, different physical location, etc. Curators have been provided necessary documentation to verify contracts, locations, hardware deployments, etc. Again, it’s all in the proposals for how the program will be run going back to KSM #35.

7

Thank you very much for the clarifications — I genuinely appreciate them.

It is clear to me that LuckyFriday operates data centers in multiple locations. However, I still don’t fully understand why the same service, for example a Polkadot Relay Chain RPC node, is being provided under two different bounties.

Isn’t this a form of duplication? Why aren’t other operators doing the same?

I also don’t understand the price difference between the two bounties for what appears to be an identical service. Even accounting for operator margins, why is the same service provided for $2,200 under one bounty and $4,000–$5,000 under another?

Finally, isn’t there a certain overlap between Bounty #31 and Bounty #50?

8

Which services are you referring to? I’m not familiar with how bounty #31 operates, costs, structure, or anything. But my general understanding is that it’s individual providers, providing single sets of services from a single location. Meaning, they do not work together in a unified way to optimize for global coverage.

The IBP is focused on global latency optimization in a decentralized way which essentially means regional slots and people being able to apply for regional slots. As the slots fill up we end up with different wholly owned providers in each region and where if any one provider goes down others in near by regions can cover the load. However, all providers are required to work together in a unified way that provides standardization to endpoints which allows us to use services like geodns, anycast, etc to automatically route an end user to their closest rpc server. This is really important for any end-user facing application because many rpc calls are synchronous. This means that, you make one call and the output of that call is used as input for another call. If it takes you 200 milliseconds round-trip per request from the rpc server and your longest string of synchronous calls is say 20 that means your application will take 4 seconds to populate and become usable. If it takes you 30 milliseconds round-trip per request from the rpc server and you have 20 calls your application becomes usable in 0.6 seconds.

It’s really not rocket science, faster = better.

In addition to all this, the providers own their own hardware, operate their own networks, own hypervisor clusters, etc. No one is hosting on google cloud, hetzner, etc. There are rules regarding redundancy, performance, etc.

I can’t speak for anyone else, but I was focused on building out this DNS system as well as trying and failing to deliver on 601. We even passed on a few system chain collator seats because I didn’t want to deal with it. Everyone kind of has their own thing in addition to the IBP. Every IBP member has been involved in the ecosystem on other levels beyond simply providing RPC services. We each have our own interests, desires, and paths, it’s not that strange.

Edit:

I should add, global latency optimization will be important in the future as we move to light clients because of how it’s possible to do the same latency optimization through decentralized networks of nodes by utilizing laws of physics to determine distance between 2 nodes on a network (or the internet) and the network as a whole coming together with consensus as to nodes physical proximity to other nodes.

It’s then possible to have automatically generated sets of nodes where when a new light client comes on to the network it can announce its location and quickly find out the nearest set of nodes to pull data from. By having this global network bootstrapped it will allow light clients to be a success and not just light clients but native storage and every other requisite and end user facing service that will be deployed in the future. (We’re also the logical service providers for any computationally verified services that will be offered in any future market places).

9

Thank you very much for your input, but I still don’t understand why LuckFriday can run a Polkadot RPC node in a bounty at a price they offered in a tender for $2,200 (supposedly including their profit), and then offer the same service in another bounty for $4,000–$5,000.

I must be missing something, but it doesn’t make sense to me.

It seems that the base pricing using AWS and Google Cloud in IBP Bounty is inflated, and in my opinion, not all operators actually provide these services using those providers or at those prices.

I imagine someone with the right expertise could verify this.

10

It seems that the Dwellir Tunisia RPC nodes have not been working for weeks on AH, the Relay Chain, and several other networks. Any idea why? @otar @erin

11

Hi there @Megadot ..

Please note that Dwellir Tunisia is not an RPC endpoint provided by the Infrastructure Builders Programme, thus maybe the people you tagged will not be able to comment on that.

Cheers!

Milos

12

Thanks! And yes, it looks like it’s indeed funded by Bounty 31

@CoinStudio @TimJanssen @Cappex @frankywild

Check this issue with Dwellir rpc in bounty 31

13

Hi @Megadot, those endpoints are not funded by Bounty 31 either.

I think those endpoints were supposed to be disabled a while ago by this pr in polkadot.js, but polkadot.js has a bit of a backlog when it comes to handling PRs.

14

As per stakeworld’s comment, those endpoints should have been disabled in Polkadot.js.

They’re currently in our LB behind our primary endpoint.

None of the services within IBP program are used in any other program.

This is made very clear in our proposals and the moderators you have tagged can verify this.

15

Can someone help me understand the new IBP Bounty top-up proposal? I still have a lot of questions.

If they were already funded by the treasury based on previous milestones with around $15,000 allocated for hardware purchases, why is hardware still being included as a cost in this proposal? Shouldn’t they only be compensated for bandwidth at this point?

And second, why are they using AWS pricing as the cost reference when almost none of them actually run their nodes on AWS?

I genuinely don’t understand why the recent Helikon proposal exposing IBP’s inconsistencies and questionable cost structure didnt pass.

A lot of the concerns raised seemed completely valid and deserved a proper discussion instead of being ignored. At this point, it feels like there’s very little accountability around how these top-up proposals are being justified.

Am I missing something here, or does this not make much sense to anyone else either?

16

Thank you for putting together this detailed review. In addition to the financial and structural points raised, I would like to add my strong concern regarding centralization. Relying on centralized infrastructure or routing introduces a critical attack vector for our RPC nodes. I highly advise anyone involved in these discussions to study the recent rsETH exploit, which specifically targeted and compromised off-chain RPC nodes. While the probability of an rsETH-style attack on Polkadot might seem low, the potential impact is catastrophic. A successful exploit of that magnitude would completely devastate network trust and could effectively drive the value of DOT to zero.

17

Could anyone clarify the technical implementation of the GeoDNS/Anycast services? Running a DNS check on polkadot.ibp.network returns zero IPv6 AAAA records. Is it only old school IPv4

18

Keep up the excellent work! It is incredibly refreshing to see this level of quality, audit-style analysis highlighting a completely different angle on our treasury proposals.
Looking into the discrepancies you’ve outlined, I really get the impression that bureaucracy has partly taken over. It feels like some of these bounties are just running on autopilot, still operating under the assumptions and budgets created back when DOT was firmly a top 10 blockchain. The points you raised regarding the AWS/Google pricing models versus the actual bare-metal hardware being used, as well as the potential for overlap, are critical.
We need exactly this kind of rigorous data auditing and oversight to ensure capital efficiency and transparency in the ecosystem.

19

If they were already funded by the treasury based on previous milestones with around $15,000 allocated for hardware purchases, why is hardware still being included as a cost in this proposal?

@Megadot thanks for following up on this. IBP members received $30,000 in total for hardware, not just $15,000. Please check rank 2 and 4 in this document.

I genuinely don’t understand why the recent Helikon proposal exposing IBP’s inconsistencies and questionable cost structure didnt pass.

One desired function of those proposals was to demonstrate the actual base cost of running these systems, and point at the ridiculous levels extraction (80-90% profit margin) being applied by IBP, facilitated by a couple of its managing members. So, we were not seeking funding solely.

And yes, it is a massive red flag that it did not spark enough discussion, at least publicly. I don’t mind the proposals not passing, they are just a “proposal” after all, but I do mind the absolute public indifference from W3F and Parity regarding what was portrayed in the proposals. It’s a pity, but it is what it is.

Thanks to the dedicated community members such as yourself, the discussion is kept alive as it should be.

I would like to add my strong concern regarding centralization.

@tom1 if it’s a centralization concern to run RPC nodes on a distributed set of cloud providers, all on bare metal, mixed with provider-run bare metal in location data centers, then seriously, what are we doing here? Are you even aware of what you are suggesting? So I guess you think a decentralized+permissionless network should have an “official, preferred, and permissioned” set of RPC providers?

Besides, the majority of Polkadot RPC interface AND validator set is already running on cloud providers. So it’s been posing a risk, and we didn’t know it?

Mind you, RPC nodes do not provide finality. Security, consensus, and finality are provided by validators, not RPC nodes. And even for validators the Polkadot wiki is not ruling out cloud providers, and actually lists a number of them with suggested configurations.

I really get the impression that bureaucracy has partly taken over. It feels like some of these bounties are just running on autopilot, still operating under the assumptions and budgets created back when DOT was firmly a top 10 blockchain.

Yes, very much indeed. So much so that it’s obscene.

I suggest the community to stop buying the “a-couple-of-us-vs-the-rest-of-the-internet” rhetoric at 80-90% profit margin. Your traffic after all is traveling through someone else’s hardware eventually - be it a cloud provider’s infrastructure, your colocation center’s infra, a Tier-2 transit provider’s routers, or the submarine cables of a Tier-1 provider.

More later.

20

@kukabi Sorry for the misunderstanding, but that is not quite what I meant.
My centralization concern has nothing to do with whether the nodes are running on distributed cloud providers or bare metal. The critical issue I am pointing to is the centralized geoDNS and load balancing infrastructure, which effectively breaks end-to-end (e2e) encryption and opens an attack vector to DDOS attacks. Also the operational risks having a single entity with so much power of the RPC nodes.
If you re-read my initial post about rsETH incident , the threat model I am highlighting is not about direct on-chain exploits or altering finality. The latest and most sophisticated attack vectors in the industry are specifically targeting off-chain infrastructure.
In our ecosystem, an off-chain exploit of this nature could easily be weaponized against platforms like Hydration or vDOT. Even if the original DOT base layer isn’t directly compromised in such an event, a massive off-chain exploit within the ecosystem would still have catastrophic implications for Polkadot’s overall reputation and user trust. I can strongly advise to study the rsETH / KelpDAO incident. In the chain of mistakes was a harmless governess change to increase LTV of rsETH to 90%.