Discover ChainQL: Unique Network's Advanced Tool for Substrate Chain Storage

Ecosystem
1

Hey fellow developers,

It’s time to unveil a tool we at Unique Network have been fine-tuning - ChainQL. This extension over Jsonnet, or more specifically Jrsonnet (its implementation in Rust), is here to take how we engage with Substrate chain storage to the next level.

At first glance, you might liken ChainQL to PolkadotJS and its API. But wait! ChainQL packs robust scripting functionality, on-demand encoding and decoding of data and keys, and efficient traversal of lazily-evaluated storage maps. If you’re seeking ease of scripting and top-notch data handling, look no further.

Sure, ChainQL is similar to PolkadotJS in core functionality, but it primarily shines as a data querying tool. While it doesn’t support signing and sending extrinsics, its unique features make it a solid contender in the tooling space. For those unacquainted with Jsonnet, or to understand its syntax better, check out the link below. ChainQL only enhances Jsonnet with the cql library (adding bigint support along the way).

The idea of ChainQL first emerged when we were in the process of migrating certain fields in our NFT structures. We found ourselves needing to calculate the exact number of storage writes this migration would require - a metric that using PolkadotJS couldn’t readily offer without some awkward and clunky workarounds. This was during the pre-try-runtime days, making our situation all the more challenging. It wasn’t long before we had a prototype on our hands - a solution that managed to deliver the required metrics with code that was succinct, compact, and pleasing to the eye.

Fast forward a few iterations, ChainQL has become our go-to tool for chain spec mutation, for both testing and production environments. Be it fetching live data from existing chains to launch a testnet, or gathering diverse metrics from our chains, ChainQL has proven itself to be a very helpful jack-of-all-trades in our toolkit.

Speed? ChainQL, with its lazily-evaluated storage and Rust implementation, goes toe-to-toe with PolkadotJS. And when it comes to the heavy lifting - data manipulation, encoding, decoding, and recording into JSON format - ChainQL doesn’t break a sweat.

Zombienet integration

Our journey with ChainQL doesn’t stop at its current features - we’re constantly exploring new frontiers. One such endeavor is to enable Zombienet to tweak its generated chain specs prior to launching the chain. This integration is not limited to ChainQL, and it would enhance our tool with additional flexibility, fostering a more dynamic testing environment.

Progress is underway; we’ve submitted a PR that’s sparking some interesting conversations. The review stage has brought up important discussions about safety guards to ensure the robustness of this feature. We’re encouraging feedback and open dialogue around this potential improvement, and the corresponding issue is open for discussion. Our team is excited to explore this development further, as we continue refining ChainQL to serve you better.

Please feel free to share your thoughts and join the discussion - your insights are invaluable to us!

Example

ChainQL is, as stems from the name, mainly a tool for querying data. It cannot replace PolkadotJS in terms of signing and sending extrinsics. As such, it can always read a chain’s storage, but one of the few ways it can write to it is by adding storage keys to a JSON chain spec.

Let’s dive into an example: Querying the system balances of another chain and integrating them into the accounts of a local chain spec.

We will modify a generated raw chain spec with data pulled from another chain. We will evaluate the chain’s accounts and add them to the genesis, and also modify the likely already existing total issuance with the combined issuance of the received accounts.

function(rawSpecPath, chainUrl)
        local sourceChainState = cql.chain(chainUrl).latest;

A Jsonnet file starts with a function declaration - this one takes the path to the spec to modify and a URL of a node of the chain the data of which we want to pull. Immediately after, we use the cql library to get the latest complete state of the chain (not yet evaluated, of course). Alternatively, we could get the state of any block in the chain.

local
	// store all keys under the `Account` storage of the `System` pallet
	accounts = sourceChainState.System.Account._preloadKeys,
	// get the encoded naming of `pallet_balances::TotalIssuance` for future use
	totalIssuanceKey = sourceChainState.Balances._encodeKey.TotalIssuance([]),
;

We create two variables: an array of all system accounts, fully loaded and evaluated this time, and the encoded key of the total issuance, by which we will access and modify the issuance later.

local accountsEncoded = {
        // encode key and value of every account under `system.account` and add them to the chain spec
        [sourceChainState.System._encodeKey.Account([key])]: 
                sourceChainState.System._encodeValue.Account(accounts[key])
        for key in std.objectFields(accounts)
};

Here we iterate over the accounts, encoding their key and value as we go. When evaluated, the accounts are automatically decoded for convenience, thus this step is necessary for putting encoded entries into the raw chain spec.

local incomingIssuance = std.foldl(
        function(issuance, acc)
                issuance + acc.data.free + acc.data.reserved
        ,
        std.objectValues(accounts),
        std.bigint('0'),
);

With this, we combine all incoming accounts’ free and reserved fields together to form their total issuance which we will add to our existing one in the next step.

Note: std.foldl’s signature is the following: std.foldl(func, arr, init) (stdlib reference). std.bigint is a feature of Jrsonnet, and the notion of bigint is absent in Jsonnet’s standard library.

local rawSpec = import rawSpecPath;

// return the modified JSON
rawSpec {
  // '+' expands a nested field with the entries inside, adding to the existing ones
  genesis+: {
    raw+: {
      // concatenate the accounts' object with totalIssuance's
      top+: accountsEncoded + {
        // encode the combined values
        [totalIssuanceKey]: sourceChainState.Balances._encodeValue.TotalIssuance(
          // decode the chain-spec's already existing totalIssuance ('super' points to the parent field)
          sourceChainState.Balances._decodeValue.TotalIssuance(super[totalIssuanceKey]) +
            incomingIssuance
        )
      },
    },
  },
}

Here we import the JSON from the file that the rawSpecPath points to, and add to the top field nested within it the fetched accounts and the total issuance. Note: we do not take into consideration the total issuance funds’ potential overlap with the testnet’s present accounts.

And that’s it! This will output a ready-for-launch chain spec with all funded accounts of an existing chain. For more examples, visit chainql’s repository on GitHub!

So, there you have it. Give ChainQL a spin and let us know your thoughts. We’re all ears for feedback and suggestions. Happy coding!

3 Likes