Polkadot-API updates thread

Hello there!

I’m Carlo, the most recent member of Polkadot-API team.

Polkadot-API (GitHub) is a project conceived to build upon and surpass the foundational achievements of the PolkadotJs project. At its essence, Polkadot-API is a meticulously crafted suite of libraries, each designed to be composable, modular, and fundamentally aligned with a “light-client first” philosophy. Our overarching mission is to empower dApp developers by equipping them with a comprehensive array of tools, pivotal for the creation of truly decentralized applications.

We’re currently on beta phase, and we plan to release our stable v1 by the end of Q2 2024. Following the example of smoldot, we’d like to open this thread and inform the community on a weekly basis (we’ll try our best to maintain the pace :wink:) a highlight of the progress made during the week.

We’re really active and responsive on our main repo and we, of course, maintain a CHANGELOG with all the relevant changes, but we seek a higher level of transparency with the community with this initiative.

Polkadot-API team (@voliva, @josep and @carlosala)

6 Likes

Lots of stuff (and some breaking changes) have been going on these week! We released v0.6.0 today (2024-05-03) wrapping all the changes to deliver them to our beloved early adopters. A big shootout to @kheops, @tim_b, and @mcornholio. They’ve been amazing reporting many bugs and unexpected behaviours. Thanks!

We want to do highlight some of them:

  • Transactions are now mortal by default. As Valentin from Nova pointed out, transactions should be mortal by default. That’s the new default! PR
  • Fee estimation is now more precise. Transaction fee depends, besides other stuff, on the length of the extrinsic. We’re now estimating better this measure, and therefore, fees are now closer to reality. Big shootout to @kheops for this report, besides many others! PR

Bilateral work with smoldot

These week we reported several issues to smoldot (#1787, #1788, #1789, #1804) and we want to thank @tomaka for the responsiveness and quickness solving all of them. As soon as it’ll be released, #1804 will have a big impact on loading times under certain conditions with PAPI and its sm-provider.

Tech deep-dive (PR #448)

PR #448 has been merged, and it’s an incredible change that makes our checksum-based method to construct types way more stable and resilient. Several things changed:

Tuple == Array?

A Tuple type that has the same inner type in all of its components (for example (u128, u128, u128)) now has the same checksum as a fixed-size array of that type (in our example [u128; 3]). These two types are encoded and decoded the same exact way, and therefore, they should hold the same checksum.

Mirrored types:

We also found another case where some circular types that were identical with each other would give different result. As an example, we had in our known-types repo XcmV2Instruction and XcmV2Instruction1, which were types that when you use them they are absolutely identical, but they had different checksums. And these types were in the polkadot relay chain. The problem is a bit hard to explain, but it’s when there were equivalent types with circular references where at some point you can switch from one to the other. This has been solved in the function that calculates the checksum.

Represented in a graph of types, we have something like this in the metadata:

image

Have a nice weekend!

6 Likes

:wave: from Soda,

We are excited about adopting the Polkadot API in our Ocelloids SDK. We have a few questions:

  • Could you please explain how compatibility is maintained between different versions of the runtime types? Do we need to generate them upon runtime upgrades, and how does this impact existing logic related to previously generated types?
  • Are there currently hashing-configuration-aware helpers available, or are there plans to develop them, to build storage keys for querying?
  • Do you have any plans to introduce high-level features such as recursive decoding and correlation of batch calls, or is this kind of functionality not within your current scope?

Thank you!

2 Likes

Hi @sodazone and thanks for reaching out!

I’m going to briefly answer your questions in here, but I would like to ask you to please move this conversation into GH, as I really think that it’s a much better suited forum for these kinds of discussions. Feel free to open as many discussions as you want.

Please have a look at this section of the docs, as I think that it already answers some of your questions.

Yes, there are. Also, some of them are already being used by other teams in the ecosystem. Although, they are not yet very well documented and/or stable, emphasis on "yet ". Eg: the getChecksumBuilder on @polkadot-api/metadata-builders and the (currently undocumented) view-builder.

EDIT: sorry, I think that you were referring to another kinds of hashing helpers. Yes, of course, please have a look at the tests of the @polkadot-api/substrate-bindings and this very straightforward storage helper. However, if what you want is to get the storage encoders/decoders for a given storage-entry, I recommend using the metadata-builders.

We already have recursive decoding, of course.

I don’t know what you mean by that. I mean, we support batch calls, of course… But I don’t know what you mean by correlation of batch calls.

I’m not sure because I don’t quite understand what you are asking, TBH.

Please, let’s move the discussion to GH :slightly_smiling_face:

1 Like

Tis’ been a pleasure!
Love the velocity and i’m excited about where the project is going.

Out of interest what would you say are the big ticket items the team wants to deliver next?

We always put at the very top of our priority list are the issues opened by the devs/teams that are using polkadot-api. Even if there is a lot of context-switching, we think that’s the right thing to do, because it helps at creating faster feedback loops, which we think is going to be very important for polishing the library for the delivery of the first stable version. Similarly, we try to spend as much time as possible in calls with different early adopters to better understand how they are using the library, what their thoughts are, the things that the docs didn’t explain properly, etc.

Then, leaving aside bug-fixing and other requested improvements, our main 3 focuses at this moment are:

1) Tx improvements:

As I explained in the last quarterly update there are a number of improvements related with the handling of transactions that are quite urgent. Yours truly is the one leading this front. I have already started tackling this, but the main things that need to be improved are: revalidate transactions after they have been broadcasted (this is one of the most important items), allow advanced users to pass custom nonce when creating a transaction, inform earlier about the events related to the transaction (as of right now you have to wait until its finalized or query those on your own), and a few more items that I will eventually put in a long issue.

2) Improve docs, jsdocs and tests:

@carlosala is the one leading this front. We think that this is a win/win, because on the one hand it helps him at getting more familiar with the top-level API, and the other hand it gives him an opportunity to check how things behave vs how things should behave, etc. There is quite a lot of work to be done in this end, but I think that we are making really good progress.

3) Improve bundle-sizes, loading times and types/DX:

@voliva is leading this end. He has already made quite a bit of progress, but there is quite a bit more work that needs to be done.

There are many other things that we would like to tackle, but we try to “pick our battles”, as they say. I mean, we also try to find some time to improve some of our inner libraries which can empower other dev-tool builders, but we think that our top priority at the moment should be to empower DApp builders.

4 Likes

Hello there! :wave:

We’re excited to announce that this week, on Friday (2024-05-10), we released polkadot-api@v0.7.0! A big thank you to our early adopters for their invaluable feedback and bug reports!

This update brings several enhancements, particularly on the transaction side, which we’ll explore in more detail in the following section. First, let’s briefly review some other changes this week:

  • Introduced a _request method in the client, providing an “escape hatch” for JSON-RPC provider consumers to access debugging endpoints. #482
  • Separated descriptor types from their values, reducing unnecessary display of empty pallets in operations. #502
  • Implemented “lazy-loading” for descriptors, significantly reducing the bundle sizes for dApps utilizing PAPI. #504
  • Enhanced the consistency and stability of the stop event recovery process. #493 #495
  • Ensured pjs-signer now returns the same payload as @polkadot/extension. #501
  • Updated smoldot to version 2.0.26, addressing issue smoldot#1804.

Transaction Improvements

This release features numerous enhancements regarding transactions!

Additions

  • We’ve introduced support for optimistic transactions. The .submit family of methods now includes an optional at parameter, allowing developers to specify the block for submission. #486
  • Added a custom nonce option, enabling advanced users to queue multiple transactions smoothly. #498

Changes

  • Refined the structure of transaction events:
    • signed: Now returns the txHash for user convenience. This resolves issue #346.
    • broadcasted: This event also now includes the txHash.
    • txBestBlocksState: This event has been significantly revised to provide more detailed information, ensuring consistency with the finalized event.
    • finalized: Continues to provide the same detailed event information.

We believe these changes will greatly enhance the functionality of our transaction API, which we expect to remain stable leading up to the official release.

Have a wonderful weekend ahead!

1 Like

Hi community! :wave:

We’ve made several improvements to PAPI this week! Let’s directly dive into the details!

Bundle Size

PAPI descriptors have seen a significant reduction in size for multi-chain dapps. Our tests show the size has decreased by just over 50% in both parsed and gzipped formats. However, for single chain dapps, there is a slight increase of about 10%.

Controlling bundle size is a core aspect of PAPI development. Over the past few weeks, we’ve been systematically researching ways to reduce it even further.

CLI Improvements

This week, we enhanced the developer experience with the papi CLI. Here are the changes:

  • Added support for fetching metadata and generating descriptors from a WASM runtime without needing a running node, chainspec, etc. This should simplify the workflow for preparing a dapp for future runtime upgrades.
  • Accelerated the papi update and papi add commands while also reducing their memory usage.

Documentation

We’ve made significant improvements to our documentation! You can find more sections now on our docs page.

Quiet work

Besides all the visible work, we’ve focused on researching several things that we think that could be improved in PAPI.
One of them is how to improve our isCompatible (see docs) check. We aim to make it easier and more straightforward for developers to ensure it behaves as expected. We’re making really amazing progress here, and you’ll hear about it sooner rather than later!

Have a great weekend ahead!

2 Likes