It is clear that an onchain data monitoring solution is super critical. I believe every parachains have built some in house monitoring solution. It will be great if we can work together to avoid duplicated work.
Especially with XCM and have all the parachains interconnected, it is not enough to just monitor data on a single parachain. All the connected parachain data are also important. For example, to ensure the reserve assets matches to XCM reserved transferred asset amount. For such monitoring, we will need to be able to pull data from multiple parachains.
In additional to that, each parachain would have its own way of storing data. For example, for token data, some parachains could use pallet-assets or orml-tokens or their own multi-token pallet. Therefore, it is hard to have a generalized way to process those data without getting to implementation details of each parachain.
It will be great if we can all work together to build a generalized multichain monitoring solution that allow anyone to easily create monitor for any parachains and reuse / composite / collaborate monitors together.
I’ll share about our project that I think could be relevant. We have been working for a while on different ways to integrate Substrate and Matrix as there are lots of benefits the decentralized communications protocol can bring to the table. One such benefits being monitoring on-chain data where Matrix can take care of publishing insights in rooms that anyone can join to monitor and get notified automatically.
Our PoC is a container with a Substrate node, a lightweight homeserver like dendrite/conduit and a general purpose appservice that comes with its own WASM runtime to extend the homeserver functionality with plugins that can be composed. One such plugins is a Substrate client that talks to the node via RPC to let other plugins submit transactions, query or subscribe to state changes. The next stage of this project is to pack everything in a single lightweight and easy to deploy binary where the WASM runtime also hosts the homeserver and a smoldot light/full node as plugins. Chains will also have a pallet-summa(substrate meets matrix) that couples the two ecosystems further adding nice extra features.
Having a general purpose WASM runtime where plugins can subscribe to events happening in a chain(or multiple) and then be able to publish the result of a computation on a storage that anyone can replicate in a decentralized way is a great solution to this kind of problem.
I briefly presented Moonbeam’s in-house monitor solution for this, and the way we set up our monitor to track Sovereign Account balances on each chain and total supply of the representation asset. You can find the slides here: 20221129 - Moonbeam XCM Sub0 [External] - Google Slides
We rely on public endpoints (with redundancy and we check for endpoint health), and programming how each asset is defined as a multilocation, and how to read the sovereign account’s balance on each specific chain.
The idea’s mentioned seem good but it might take some time to implement.
Wanted to let you know that we have built a multi-chain monitoring SDK that addresses some of the use cases that you’ve mentioned i.e. connection with Smoldot and pallet-specific modules. We also have an example application for monitoring sovereign account assets across parachains.
You can find all the details in our post, including the Github repo link and our future plans on building a potentially decentralised network of monitoring agents. Would love to know if what we are building is useful for you and what capabilities you would like to see in a multi-chain monitoring solution.
I see it is an SDK which could be a useful foundation component of the monitoring solution I want, but lot more works are still required for it to be useful for my use cases.
You’re exactly right that it’s simply a foundation piece of a bigger solution. We’re currently fleshing out a proposal for a follow-up grant to build the portal, where users could deploy and manage pre-defined configurable monitoring programs, which would be made available in a marketplace. These programs will be built with our SDK and will contain the complete logic for complex monitoring use cases. They can be instantiated as monitoring agents that will emit data streams that we plan to make it easy to integrate with existing monitoring, alerting and visualisation infrastructure e.g. Prometheus, Grafana
The proposal is still WIP but we already have some mockups of the portal available here:
It would be great to hear from you what kind of use cases you’re looking at so we can take it into consideration when designing the system.
Ensure consistent balances of sovereign accounts. We are usually using reserved transfer to transfer assets between relay/parachains. This creates an invariant that the total issuance of the foreign asset on dest parachain should not be more than the reserved asset amount of the sovereign account of the dest chain on source chain. We should receive alerts if a difference is detected for more than a few minutes.
Ensure other crosschain state consistency. Acala/Karura offers liquid staking solution and the parachain controls multiple staking accounts on relaychain. The invariant is that the total value of the liquid staking asset on parachain should match to the staking assets on relaychain.
We do have an example application for your first use case. You can check out the code here. Of course, it’s still missing the part of time series data and alerting which is what we plan to tackle in our next grant.
When we have the time series and alerting implemented, and the applications as one-click deploy agents, would this be something that you and your team would be interested in using?
We’ve rolled out the first release of the XCM Monitoring Server that meets most of the points above. Also supports connecting with smoldot. We plan to add open observability in the future.
Let us know if you have any questions or feedback!