I’d like to share that the new fork-aware transaction pool implementation for substrate-based nodes is ready and available for testing. This improvement addresses several key issues with the previous transaction pool, which often led to stuck or dropped transactions.
What was wrong with the old implementation?
In short, the previous transaction pool was providing an invalid ready set of transactions to the block builder, causing valid transactions to be dropped on collator. This, in turn, resulted in transactions being stuck on the RPC nodes.
This issue is easy to reproduce locally in a small network (e.g., 1 RPC node, 2 collators). After submitting a large number (3-4 blocks) of transactions from a single account, some of the transactions would never make it into a block, becoming stuck on the RPC node. (I can share more details and setup instructions if anyone is interested).
More techincal details about the root cause can be found here and here.
How the fork-aware transaction pool fixes this?
The new implementation maintains a transaction pool state for every fork, meaning it keeps a ready set of transactions for each block that’s notified to it. This ensures that the block builder always works with the most up-to-date ready set of transactions, aligned with the state of the given fork. No future or stale transactions are provided to the block builder.
For a deeper dive into the internals of the transaction pool, you can check out an initial proposal and also this document.
What’s next?
While the new implementation is ready for testing and should resolve the major issues, there are still areas for improvement. A list of known future work and enhancements has been documented, and I’d appreciate your input on prioritizing these.
Feel free to check the planned future work here. If you have specific priorities, please let me know or give a thumbs-up on the GitHub issues related to the future work to help prioritize what should be tackled next.
Grafana
I’ve also set up a Grafana dashboard with insights into the new transaction pool mechanics. You can view it here.
If anyone is interested, I can export and share the dashboard JSON.
How to run it?
The PR description is providing a code snippet that needs to be integrated with the custom node. For omni-node the fork-aware transaction pool is already there.
To enable new pool simply use the following command line argument: --pool-type=fork-aware.
For debugging purposes the following log settings will be helpful: "-lbasic-authorship=debug -ltxpool=debug".
Thank you all for your support and patience throughout this process. Your feedback and testing will be invaluable as we continue refining the transaction pool. If you encounter any issues or have suggestions, don’t hesitate to reach out. I’ll be glad to help.