Submerge Crystal: Open-Source Indexer API for Polkadot SDK Chains

Tech Talk
1

Introduction

This thread will be used to share Submerge release updates and news with the Polkadot community, starting with Submerge Crystal, the core indexer component and its API. Submerge is an open-source (GPLv3) indexing, data processing, and compliance platform for Polkadot SDK-based blockchains, being built by Helikon and funded by the Polkadot treasury.

Submerge Crystal API

Submerge Crystal is the core indexer component of Submerge. It is deployed per-chain, and its API exposes blocks, extrinsics, calls, events, storage traces, genesis state, and runtime metadata as queryable REST resources.

Crystal indexes the base data consumed by Submerge’s upper layers, and is also a standalone indexer API for developers who need direct, structured access to historical chain data.

Crystal API documentation is available at docs.submerge.io/crystal-api. The API is currently live for Polkadot Coretime and Polkadot People. The complete set of Polkadot and Kusama relay chains and their system parachains is currently indexing, with public endpoints to follow in the coming days.

The Crystal API is organized around seven resource types:

  • Blocks
  • Calls
  • Events
  • Extrinsics
  • Genesis
  • Metadata
  • Traces

Below is a summary of what each of these sections provides.

Blocks

The Blocks section exposes the chain’s block history through two endpoints. Blocks can be listed and filtered by author, status, runtime spec version, and block number or timestamp ranges, or queried directly by hash or number. Pruned blocks are marked through the status field, alongside finalized blocks.

Calls

The Calls section provides access to the chain’s runtime call history through ten endpoints. The API indexes every dispatched call as a first-class entity, not just as a member of an extrinsic.

This means a query like pallet_name=broker&call_name=renew returns every call instance across the chain history, regardless of whether the call appeared at the top level of an extrinsic or was wrapped inside a call such as utility.batch, proxy.proxy, multisig.asMulti, etc. Name filters accept partial and case-insensitive matches.

The call tree is also accessible. Parent and child relationships are queryable, so a nested call can be traced back to its root call, extrinsic, and signer.

Events

The Events section exposes all events emitted by the runtime during block execution, queryable by pallet name, event name, and ranges of block number, timestamp, and spec version, or scoped to a specific block or extrinsic.

Decoded event arguments are available on demand via the include_args parameter, keeping list responses lean while making full event payloads available.

Extrinsics

The Extrinsics section provides four endpoints that expose signed and unsigned extrinsics, queryable by signer and signed status, or scoped to a specific block.

Genesis

The Genesis section provides access to the chain’s genesis storage records, with each record resolved to its pallet and storage item. This is useful to track storage changes where a storage entity begins its life in the genesis rather than initiated through an extrinsic post-genesis.

Metadata

The Metadata section exposes every indexed runtime version’s metadata, available either as raw SCALE bytes or pre-decoded JSON, and drillable down to the pallet level: calls, events, errors, constants, and storage items can each be queried per pallet, per spec version. This provides convenience for comparing a pallet’s API surface across runtime upgrades.

Traces

The Traces section provides access to storage traces. Every read, write, and delete applied to the chain’s storage during block execution can be queried. Pallet and storage item information is decoded within each trace record.

Traces are filterable by storage key prefix and key parameters, so questions such as “every write to System.Account for this account” or “every change to Staking.Bonded” become single API calls.

Querying storage traces through a public API is not common in the Polkadot infrastructure space. This is typically only available by running a private archive node with State.TraceBlock RPC call exposed.

Try It

Submerge Crystal API documentation is available at docs.submerge.io/crystal-api, with two chains currently available at endpoints as documented on the site:

  • https://coretime.polkadot.crystal.api.submerge.io/v1
  • https://people.polkadot.crystal.api.submerge.io/v1

Please give it a try and let us know about any issues or questions in this thread, via direct messages, or as a GitHub issue.

:robot: Also feel free to point your agent at the OpenAPI specification as a starting point.

Submerge is still under heavy development, so please expect major changes, and follow this thread or the GitHub repository for change tracking.

Known Issues

Filtering calls and events by pallet or name can be slow on rare matches. This is being resolved, and the fix will be patched in the coming days.

What’s Next

  • Polkadot and Kusama relay chains and the remaining system parachains will be available in the coming days.
  • A separate documentation for self-hosting Crystal will be shared.
  • Design and development work continues on Submerge’s upper layers and web application.

All updates will be posted in this thread as they become available.

Thanks for your time.

2

Hi Helikon team,

We’re evaluating Submerge Crystal as the indexer layer for a governance platform we’re building on Polkadot. The core read patterns we need are historical voting events, referenda lifecycle events, bounty events, and treasury activity, all queried by pallet, signer, and time range. Crystal’s REST surface looks like a clean fit, but before we commit to it in our technical spec we need clarity on a few things.

Chain coverage and roadmap

The current API spec lists preview deployments for Polkadot Coretime and Polkadot People only. Our use case lives on the Polkadot relay chain (OpenGov referenda, conviction voting) and Asset Hub (treasury, bounties post AHM). What’s the timeline for relay chain and Asset Hub coverage? Is Kusama on the roadmap as well, since that’s our canary deployment target? Are there plans to index parachain runtimes beyond the system chains?

Self hosting

Crystal is GPLv3, which is great. Are there public docs for running a self hosted Crystal instance, including hardware requirements, sync time from genesis, and operational guidance? We’d likely run our own instance for production rather than rely on the public API. Is there a Docker image or deployment reference we can point to?

Rate limits and tiers

The public API is capped at 5 requests per second. Our scoring, when computing scores across a 90 day window of active voters, will exceed that comfortably. Are there paid tiers or higher rate limits available for production workloads? Is the rate limit per IP, per API key, or something else?

Historical depth and reorg handling

How far back does the indexed history go on each preview deployment? When chain reorgs happen, how does Crystal handle previously returned blocks marked as proposed becoming pruned? Is there a recommended pattern for clients that need finalized data only?

Runtime upgrade handling

Polkadot ships runtime upgrades that change pallet structure and event schemas. How does Crystal handle metadata version transitions, and is there any client side work required when a new spec version lands?

WebSocket or streaming

We noticed the current API is REST only with cursor pagination. Are there plans for a WebSocket subscription endpoint or push based delivery for new blocks and events? For our dashboard layer we currently combine PAPI live subscriptions with indexer reads, but a single source would simplify things.

Missing endpoints

A few queries we’d want that don’t appear in the spec, would these fit your roadmap? Aggregate event counts grouped by signer over a block range. A combined endpoint to fetch an extrinsic plus all its events plus all its sub calls in one request, since the current pattern requires three separate calls. Filtering events by argument values, for example all balance transfers to a specific recipient.

Support and SLAs

For production use, what kind of support contract or SLA is available? Who do we talk to about a commercial arrangement?

Happy to share more about the use case if useful context. We’re trying to lock our indexer choice within the next few weeks, so any guidance on timelines would be much appreciated.

cheers,
Jose @ Two Pebbles Ventures

3

Hi @Onepebble,

Thanks for your interest.

What’s the timeline for relay chain and Asset Hub coverage?

Both targeting next week of March 4th. And Polkadot Bridge Hub & Collectives will go live tomorrow on top of the Coretime and People deployments.

Is Kusama on the roadmap as well, since that’s our canary deployment target?

Yes, Kusama will follow the Polkadot deployment. They’ll be announced starting next week too.

Is there a Docker image or deployment reference we can point to?

Self-hosting documentation will be published very soon. Docker image has already been developed, but we need to improve the ergonomics a little bit more. I’ll post updates under this thread. Hardware requirements, sync time and storage approximations would be documented over time too.

Are there paid tiers or higher rate limits available for production workloads? Is the rate limit per IP, per API key, or something else?

Requests are currently limited per IP, as you noted, at 5 requests per second. Paid tier and other income models are under consideration for sustainability, we will publish them as soon as the work with the base layer gets completed.

How far back does the indexed history go on each preview deployment?

The complete history is stored.

When chain reorgs happen, how does Crystal handle previously returned blocks marked as proposed becoming pruned?

Crystal stores all proposed blocks. When a slot (block number/height) gets finalized, the other candidate blocks get marked as pruned, and they remain queryable.

Is there a recommended pattern for clients that need finalized data only?

This is a good suggestion. We can add a switch to Crystal to discard all pruned blocks and related data. I just created a GitHub issue for it. But blocks are queryable by status (proposed, finalized, pruned) already.

How does Crystal handle metadata version transitions, and is there any client side work required when a new spec version lands?

Crystal handles runtime upgrades very smoothly, and assuming the chain is running on metadata version 14 or later, no extra work is required. All metadata versions and all of their components (pallets, calls, events, etc.) are also queryable through the Crystal API.

Are there plans for a WebSocket subscription endpoint or push based delivery for new blocks and events?

Yes, there will be push-based delivery for new blocks and events. We are currently evaluating WebSockets (WS) and Server-Sent Events (SSE), currently closer to SSE. This is on the immediate roadmap, likely to be delivered in the next two weeks.

A few queries we’d want that don’t appear in the spec, would these fit your roadmap?

There are higher-level components of Submerge (Fractal, Bloom) that will enable queries such as aggregate event counts by a certain property (e.g. signer) over a block/time range, or tracking the historical evolution of certain entities (accounts, referenda, bounties, etc.). These components will be published once the base layer is completed. They all introduce additional storage requirements. Therefore the capabilities would probably be evaluated on a per-case basis, starting with more widely useful ones, such as balance transfers, as you pointed.

For production use, what kind of support contract or SLA is available?

We would be happy to provide dedicated instances under an SLA contract. Let’s be in touch for it.

Thanks a lot!

4

Submerge Crystal API v0.1.1 is now available for Polkadot Bridge Hub, Collectives, Coretime, and People. Documentation is available at docs.submerge.io/crystal-api (sometimes a refresh is necessary to view the changes), and OpenAPI v3.1.0 specification is available here. Current endpoints:

  • https://bridge-hub.polkadot.crystal.api.submerge.io/v1
  • https://collectives.polkadot.crystal.api.submerge.io/v1
  • https://coretime.polkadot.crystal.api.submerge.io/v1
  • https://people.polkadot.crystal.api.submerge.io/v1

Polkadot relay chain and Asset Hub currently indexing, targeting next week. Kusama deployment, platform-specific clients, and applications to follow.

PS. This can still be considered as an early alpha release, with ongoing performance and ergonomics improvements. Please post under this thread or reach out via DM for any issues or feedback.