Dune Analytics Style Data Service for Polkadot / Kusama

It’d be nice to see something like dune.xyz but for Substrate / Polkadot.

This would be very useful to attract active power-users to the ecosystem and for analysts to follow activity in the network.

https://explorer.subquery.network/ and https://app.subsquid.io/ both have backend services which are queriable with GraphQL, but it’d be nice to have the front-end with shareable dashboards as well.

9 Likes

That’s a great idea!

We are currently in the process of gathering all the data we can for the Polkadot ecosystem and store it in Cloud Storage/BigQuery, in a format that doesn’t require running an archive node.
Our goal is to work with the builders within the ecosystem to define a common dataset and data model that we can share for interactive querying use cases like Dune. It’s also important to make sure that the quality of the data is high and that we explain how that data was gathered and provide those scripts too.

Ultimately, sharing that data will be a first step, adding an interactive query engine on top and make it publicly accessible will be the logical next step. One possible scenario would be to get that data integrated into Dune as well?

I’m open to suggestions which path forward would make the most sense. I do believe that with the scale of the data we can create a public facing analytics application that doesn’t use too many resources, we would need a bit of help with the frontend side of things though :slight_smile:

1 Like

@rphmeier completely agree. FWIW my understanding is that Dune Analytics haven’t incorporated Substrate data yet because of the data engineering challenges posed by the variations in pallets across different Substrate chains. As a result they have chosen to prioritise other ecosystems first.

I have often thought a lower hanging fruit before convincing Dune could be https://app.web3go.xyz, which are Polkadot ecosystem native and already have some Substrate data on there (for example Subquery crowdloan data). However, Web3go could benefit from having a more comprehensive data selection on there (we have communicated this).

As @Karim said we are working towards common datasets and data models for substrate analytics. Will keep front-end options in mind as we progress on this.

2 Likes

Parachain TVL - DefiLlama ← interesting :slight_smile:

4 Likes

Has anyone started work on this yet?

I would be more than happy if the desired project could use our part for NFT insights and queries they can build it. We already have some essential insights within the app for retail users, and developers are using it to build fandom/storefront shop on top of KodaDot infrastructure; we would be more than happy co-op with someone on this.

In another case we have a strong interest built it also on our own as separate service, so we rather support someone as our capacities are limited on this

Here is our KodaDot API called Uniquery GitHub - kodadot/uniquery: Client-based SDK for Polkadot NFTs and we will roll few more examples on https://developers.kodadot.xyz more about it.

1 Like

Hey @yangwao, I’d be more happy to get in touch and discuss how we can collaborate on this one. We’ve made a lot of progress with multiple teams within the ecosystem but I don’t believe we have met yet.
I just followed you on Twitter, we can discuss in DM’s there and setup a first call. What do you think?

3 Likes

There is “Dotsama Insights” https://dolpha.com/ which is showing some nice potential.

2 Likes

@alice_und_bob Do you know if they plan to make it available for anyone to create analysis like Dune Analytics?

This is exactly what we are working on at Tokenguard - an on-chain analytics tool dedicated to the Substrate/Polkadot ecosystem with no-code/noSQL approach. Check out https://app.tokenguard.io/

We are currently working with Astar and Aleph Zero to provide core metrics for these ecosystems in the form of dashboards and easy to use interface.

In addition, in the near future we plan to index and add Relay Chains and more Parachains (we are in talks with them) to finally cover the entire Polkadot ecosystem, and later Kusama.

If you have any questions or suggestions on how we could improve our product, we are open to discussion.

We (Singular.app) have to build our own solution for our NFTs, just because Dune doesn’t support Moonbeam. We would happily paid for a third party service that indexes most common datasets on Polkadot parachains (including evm ones), that has as flexible querying as Dune has, or atleast close to it. And I’m sure we are not alone

Hi @Karim do you have any update on this Cloud Storage/BigQuery?
Sorry if you already posted it in another channel. Thank you :slightly_smiling_face:

1 Like

Hi @Valentina ! For sure, much has happened since then. We worked closely with Google & Colorful Notion to get Colorful Notion’s datasets available for everyone. Here is the official announcement with some more details:

If you want to get started right away, this documentation should be helpful, tagging @sourabhniyogi and @mkchungs since they’re the masterminds behind that.

We can also get you access to our Dotlake (data lake maintained by the Parity Data team) or export some data for you to query. Whichever works.

Don’t hesitate to reach out directly also, we’re happy to help :raised_hands:

2 Likes

As the Polkadot Data Bounty has been put on hold while decentralization activities take place, we submitted this:

so as to keep substrate-etl + the Polkaholic.io indexer going for a little while longer. Here is the news in case you missed it:

The thesis is that lots of people can build their own Dune-style dashboard with decoded Substrate data and we are working on doing something suitable for hybrid Substrate+EVM chains since the high volume DAUs went from just Moonbeam+Astar (4 chains) to over 12 now and might even be most next.

We are grateful to @karim and @pavla for encouraging us to work on this dating to a year ago. We have found some decent partners on the UI side of things beyond Apache Superset to do the Dune Analytics service, but we would welcome others so we can focus on data alone.

3 Likes

This is one of the clearest yay votes I have seen. With this done, can’t we just get Dune to integrate it to their platform too?

As #248 passed (thank you all your support @pavla @karim @giotto @michalis @birdo and a couple of dozen voters), we found the right person @ Dune to work with and posted our OpenGov proposal:

substrate-etl 2.0: Polkadot + Kusama Data on Dune (12 months, 3/15/24-3/14/25)

Here is the first dashboard:

https://dev.dune.com/mkchungs/polkadot-staking-validator

We will be working with Dune over the next 3 months to systematize integration.

2 Likes

Hell yeah!