Substrate-etl.polkadot.xcmtransfers dataset + xcmv3 "context-matching"

1

We have released Polkaholic.io’s XCM Transfers Dataset in BigQuery in a new “substrate-etl” project, within a “polkadot” and “kusama” dataset. This means anyone can literally do queries like
select * from substrate-etl.polkadot.xcmtransfers;
select * from substrate-etl.kusama.xcmtransfers;
and do all kinds of analysis – We wrote up a report catering to data analysts here:
Polkaholic.io’s 2022 XCM Transfers Now in BigQuery Public Datasets: substrate-etl.polkadot.xcmtransfers | by Colorful Notion | Jan, 2023 | Medium
We did a lot of stitching together of extrinsics, events, and traces of XCM via UMP/DMP/HRMP/… to make this work. With xcmv3 merged (yay!), we hope much of this stitching work becomes unnecessary with judicious use of context-matching with XcmContext

We are looking for feedback:

  1. specifically on the BigQuery xcmtransfers dataset: for “you missed a spot here on this chain / extrinsic / …”, “what does this field mean”, “how did you handle this weird extrinsic” detailed questions that get us to improve this dataset to be as high quality as possible.
  2. how chains should coordinate matching + analytics using xcmv3’s new XcmContext to make high quality data XCM analytics as simple as possible, where XcmContext will appear right within the “xcmtransfers” data set in ways that can be filtered, grouped etc. by the community

I think with just a little bit of coordination we can use XcmContext well, making our stitching easier or downright trivial, resulting in super robust analytics. Can we outline a vision for everyone to use this XcmContext well?

5 Likes
3

@raag-bitwave It proved to be too difficult to support xcmtransfers fully across the ecosystem well especially with a systematic lack of support for comprehensive cloud resources, and hostility to RPC node providers (that we depend on). It became clear in 2023 that Polkadot tech would incorporate smoldot, XCMv3 topics and XCMP is coming soon. In the end the vanity metrics of ($/# of XCM Transfers) were not worth spending Treasury-supported resources and were well covered by others.