Production EVM RPC Infrastructure on Asset Hub (pallet-revive): Reliability Bottlenecks, Alternative Endpoints, and Best Practices

Tech Talk
,
1

Summary

We recently launched EasyCrypto on Polkadot Asset Hub Mainnet EasyDot [https://polkadot.easycrypto.fans/\] , leveraging pallet-revive for EVM compatibility. While the developer experience of compiling Solidity to PolkaVM has been solid, we are hitting severe infrastructure bottlenecks with the default public EVM RPC (https://eth-rpc.polkadot.io/).

We are opening this topic to discuss the current state of EVM RPC infrastructure for Asset Hub, explore alternative public/commercial endpoints, and understand the recommended production path for dApps relying on standard Web3 tooling.

Project & Technical Context

  • Application: EasyCrypto (https://polkadot.easycrypto.fans/)
  • Target Network: Polkadot Asset Hub EVM (Chain ID: 420420419 / 0x190f1b43)
  • Stack: React, TypeScript, ethers.js (v6), Solidity smart contracts compiled to PolkaVM via pallet-revive.
  • Primary RPC Endpoint Used: https://eth-rpc.polkadot.io/

The Problem: Public RPC Instability Under Production Conditions

Even with low-to-moderate end-user traffic, we are experiencing recurring issues that degrade the core application experience:

  1. Aggressive Rate-Limiting & Timeouts: Pointers to https://eth-rpc.polkadot.io/ frequently return HTTP 429 (Too Many Requests) or drop connections during sequential reads.
  2. CALL_EXCEPTION with Null Data: Routine read operations (eth_call) intermittently return CALL_EXCEPTION with data: null instead of resolving or returning a decoded revert reason.
  3. Absence of Dedicated Commercial Providers: Major Web3 node providers (Dwellir, OnFinality, QuickNode, Alchemy) do not yet expose dedicated public endpoints for the Asset Hub EVM wrapper (pallet-revive-eth-rpc) on Mainnet.
  4. Why Light Clients (Smoldot / PAPI) Aren’t an Immediate Fix:
    We evaluated embedding Smoldot + PAPI directly into the frontend. However, our end users connect using standard EVM browser wallets (MetaMask, Rabby), and our application layer relies on standard EVM tooling (ethers.js v6, JSON-RPC eth_* methods, and eth_getLogs for event filtering). Because Smoldot does not expose a client-side HTTP/WS eth_* RPC server, a dedicated backend RPC endpoint remains essential.

Current Mitigations (and Their Limits)

To keep the dApp usable, we implemented:

  • Client-side exponential backoff retries on RPC calls.
  • A self-hosted Nginx reverse proxy with caching for static reads to prevent CORS issues and reduce outgoing traffic.

Despite these mitigations, the single public RPC remains a critical single point of failure.

Questions for the Community & Core Team

  1. Alternative Endpoints: Are there secondary public or community-run RPC endpoints running pallet-revive-eth-rpc on Asset Hub Mainnet that developers can use as fallbacks?
  2. Commercial Support: Are node providers (e.g., Dwellir, OnFinality) actively onboarding the pallet-revive-eth-rpc layer for Mainnet Asset Hub? Who can we contact to get access or sponsor dedicated capacity?
  3. Developer Whitelist / Tiered Access: Is there an active application process or whitelist for teams with live Mainnet contracts to obtain higher rate limits on eth-rpc.polkadot.io?
  4. Recommended Self-Hosted Architecture: If running our own instance is the only production-ready route today, what is the recommended minimal setup? Running a full archive node + eth-rpc in AWS easily exceeds $600/month. Are there optimized configurations (e.g., specific pruning flags that preserve eth_call and eth_getLogs functionality) to run this cost-effectively on bare-metal?

We would appreciate any insights, shared experiences from teams deploying on pallet-revive, or contacts from infrastructure teams supporting this layer.

2

Thank you for bringing up this issue again. Unfortunately I too have experienced continuous usability issues with that rpc (not great for dev onboarding). In the meantime maybe you can use the rpc mentioned here: Reliable ETH RPC for Kusama Assethub - #2 by RustSyndicate

3

Thank you for following up and validating this issue! It’s reassuring (though unfortunate) to hear others are encountering the same friction on the dev onboarding side.

Just to clarify: our dApp is deployed on Polkadot Asset Hub Mainnet (Chain ID 420420419 / DOT), whereas the linked thread covers Kusama Asset Hub. We specifically need a reliable ETH RPC endpoint for Polkadot Asset Hub that doesn’t suffer from the strict rate-limiting, dropped connections, and CALL_EXCEPTION errors currently affecting https://eth-rpc.polkadot.io/.

4

Okay sorry I missed that. I am not aware of any at this time but will be following the discussion as it will be relevant for me as well.

5

I spent some time reproducing this against mainnet. Short version: your chain ID is right, the problem is real, and it is not something your client can fix. Reproduction tests are in a small vitest suite so anyone can re-run these numbers.

The failures are systemic

I sampled both documented mainnet endpoints back to back, one request each every 5 seconds for 15 minutes — 180 samples each, paired so a problem on my end would show up on both:

eth-rpc.polkadot.io                  : 115/180 ok  -> 200×115 404×25 429×40
services.polkadothub-rpc.com/mainnet : 180/180 ok  -> 200×180

Roughly a 36% failure rate on the primary endpoint while the other answered every sample. Your retries and your Nginx cache are not the problem, and nothing about your request pattern will fix this.

What the 429 actually is

HTTP/2 429   content-type: text/plain
Too many connections. Please try again later.

That wording is not a proxy’s invention. jsonrpsee-server builds exactly this response (too_many_requests() in transport/http.rs — 429, text/plain, no Retry-After) whenever ConnectionGuard::try_acquire() fails, and eth-rpc feeds that guard from --rpc-max-connections, default 100. It is not --rpc-rate-limit, which answers with a JSON-RPC -32999 "RPC rate limit exceeded" instead. My best reading is the eth-rpc process itself at its connection cap, though I can only see the string from outside, not which hop sent it.

For plain HTTP, jsonrpsee holds that permit per request and releases it when the response completes; only WebSocket upgrades keep it for the life of the connection. So the flag acts as a cap on concurrent in-flight requests — which would explain why nothing about your connection handling moves the error rate, and why raising the cap or adding instances should.

The 404s I can’t pin down from outside. 404 page not found is Go’s stock http.NotFound text and jsonrpsee has no 404 response at all, so presumably something Go-based in front answers once things degrade. That part is inference; the 25 counted 404s are measurement.

Worth knowing for your retry layer: ethers v6 already retries 429s on its own. It keys off the status code alone — the text/plain body is irrelevant — and backs off exponentially up to 12 attempts, honouring a numeric Retry-After when present. A sustained 429 window therefore reaches your code late, as a SERVER_ERROR or timeout, and any backoff you add stacks on top of ethers’. The 404 throws immediately.

One thing I got wrong, in case you try it

My first read was that HTTP keep-alive would sidestep a connection limit, and I had a clean 300/300 run to back that up. Ordering bias — whichever pattern ran first met a less-loaded period. With alternating order and counted sockets:

keep-alive 111/160   churn 130/160

These failures are bursty and heavily clustered, so I would not read much into that gap in either direction — but one round returned 40 consecutive rejections on a single socket, which the per-request permit explains: the guard is not counting your connections, so reusing one buys you nothing. Keep-alive and batching are still worth doing on general principle (batches of 3, 60 and 200 all work, and wss:// works on both endpoints), just don’t expect them to move your error rate.

On the CALL_EXCEPTION with null data

Three different things produce this, and only one is arguably node-side:

  • A revert that carries data is reported correctly — code 3, message execution reverted: <reason>, payload in data. ethers decodes it fine. If you are seeing null data, it is not this.

  • Failures with no revert payload (out of gas, nonce problems, decode failures) return code -32000 with data absent — there is no payload to return.

  • eth_call to an address with no contract code returns "0x", not an error, and ethers v6 fails to decode the empty return. Reproduced end to end against mainnet:

    ethers code=BAD_DATA data=null

    Ethereum behaves the same way.

The explanations that would have pointed at the node don’t hold up: eth_call at head−1,000,000 works (state is archival, not pruned), eth_getLogs across a 100,000-block range returns with no cap error, and precompiles work.

Given the 36% endpoint failure rate, my guess is most of your “intermittent” call failures are the 404/429 windows rather than anything about your contracts. Log the HTTP status alongside the ethers error and you should see it quickly.

The second endpoint

The docs list two mainnet RPC URLs, not one:

  • https://eth-rpc.polkadot.io/ (ParityOps)
  • https://services.polkadothub-rpc.com/mainnet/ (OpsLayer)

Same stable2603 release line (the builds differ — 965e247 and 241bd90), same chain ID, very different availability in my measurements. Worth configuring both with failover.

On third-party providers

Your read matches what I could check. Tatum and LuckyFriday answer -32601 Method not found for eth_chainId — plain Substrate RPC, no eth-rpc proxy. The OnFinality, Blast, IBP and Dwellir EVM hostnames I tried either do not resolve or sit behind an API key (Dwellir: 403 API key does not exist, so I cannot see what is behind it). None of the endpoints I could reach provide eth_* for Polkadot Hub — weaker than “no provider offers this”, but it still leaves you without a public fallback. That, plus the availability numbers above, seems like the thing worth escalating.

6

gm, shield.markets is hosting some public eth rpc’s::

8

Thank you so much for hosting these public RPC endpoints! We just tested polkadot-assethub-rpc... on our dApp and it runs smoothly (~300ms latency, zero rate-limit issues). Great work by shield.markets for supporting the ecosystem! :rocket:

Also, if you have any dApps or projects looking for cross-promotion or launch opportunities, we’d love to collaborate — we have a Launchpad section ready in our dApp to feature promising projects.

Let’s connect if you’re open to synergies! :handshake: