We have released Polkaholic.io’s XCM Transfers Dataset in BigQuery in a new “substrate-etl” project, within a “polkadot” and “kusama” dataset. This means anyone can literally do queries like
select * from substrate-etl.polkadot.xcmtransfers;
select * from substrate-etl.kusama.xcmtransfers;
and do all kinds of analysis – We wrote up a report catering to data analysts here:
Polkaholic.io’s 2022 XCM Transfers Now in BigQuery Public Datasets: substrate-etl.polkadot.xcmtransfers | by Colorful Notion | Jan, 2023 | Medium
We did a lot of stitching together of extrinsics, events, and traces of XCM via UMP/DMP/HRMP/… to make this work. With xcmv3 merged (yay!), we hope much of this stitching work becomes unnecessary with judicious use of context-matching with XcmContext
We are looking for feedback:
- specifically on the BigQuery xcmtransfers dataset: for “you missed a spot here on this chain / extrinsic / …”, “what does this field mean”, “how did you handle this weird extrinsic” detailed questions that get us to improve this dataset to be as high quality as possible.
- how chains should coordinate matching + analytics using xcmv3’s new
XcmContextto make high quality data XCM analytics as simple as possible, where XcmContext will appear right within the “xcmtransfers” data set in ways that can be filtered, grouped etc. by the community
I think with just a little bit of coordination we can use XcmContext well, making our stitching easier or downright trivial, resulting in super robust analytics. Can we outline a vision for everyone to use this XcmContext well?