160 Dials: How Two go-libp2p Defaults Halved Our View of Polkadot

Tech Talk
1

Hi folks,

I wanted to share our newest blog post on Polkadot numbers. We took the time to overhaul our crawling infrastructure after funding ran out last year to look into the flakiness of our Polkadot results. It turns out that many Polkadot nodes have an issue with pruning old listening addresses. This impedes connectivity within the network. We believe this is not intended and would urge the networking team to look into this before the issue becomes prohibitive for the connectivity among nodes (it is to a large degree already).

The full blog post is here: 160 Dials: How Two go-libp2p Defaults Halved Our View of Polkadot | ProbeLab Analytics

But the most important finding for Polkadot is probably that 50% of nodes advertise >26 listening addresses and 10% advertise over 90. When another node tries to connect to such node it will need to try all of these addresses sequentially (8 in parallel by default in go-libp2p, not sure about litep2p). Because of timeouts you often won’t be able to go through the entire list. If the current listening address is at position 100 but you only get to dial 16 addresses (10s dial timeout with a 5s TCP connect timeout, and 8 parallel dials), then you can’t connect although there is a good address in the list.

Check out this Polkadot peer for example: Sonar — 12D3KooWGWAkLkoEZZETq6W… | ProbeLab Analytics

It has a ton of addresses registered in the DHT. It would be great to hear what’s going on.

Best

Dennis from ProbeLab

2

Interesting find. Stale listen addresses piling up in the DHT record would be consistent with the node not expiring old external address candidates, so it may be worth checking whether it only happens behind NAT or with changing IPs. A cap on the number of advertised addresses, ideally preferring recently confirmed ones, would limit the damage even before the root cause is fixed. Do you know if the litep2p nodes show the same distribution as the libp2p ones?

3

I don’t know the litep2p/libp2p distributions but I can see in the data that it’s mostly the Mosaic Chain Collator who’s showing the problematic behaviour