Polkadot Native Storage Updates

Tech Talk
41

@somedude, lets use the discussions here GitHub - eigerco/polka-storage: The Polka Storage Parachain Project

1 Like
42

Great. Just fix that 404 [1] in README.md and you’ll be golden!

The Web site link in repo metadata is also 404 (URL of the same Web site which seems to be the place for docs).

I don’t know what’s supposed to be on the site, but I hope you’ll have a getting started walk-through, so that those who aren’t experts can at least put and get some data without too much pain :slight_smile:

[1]

Check out the book at https://eigerco.github.io/polka-storage/getting-started/index.html.
43

Our apologies!

While we fix it, you can check the book here:

2 Likes
44

For anyone looking to try, this is almost ready, but if you’re not a Rust developer-level troubleshooter it’s probably better to wait another 1-2 weeks.

Some sharing for those looking to try right now (Nov 24, 2024):

  • Errors exposed to the user are not informative (WIP already, but not done yet) - see the example below
  • The docs are still being improved and, in the current iteration, may be confusing (in short, installation/setup workflow is confusing, but also the docs are out of sync with the repo, which made me waste 1-2 hours until I discovered that). This will be improved as well
  • Binaries/containers are out of sync with the docs as well (e.g. the faucet command)
  • The repo defaults to develop but the docs refer to releases and some PRs with fixes don’t have useful recent commits, while latest release (and therefore, containers) has a 3 week old release). Not unexpected, but it may be hours until you figure all this out.

The good news is it’s looking promising and while I spent hours trying to get past the error below (and couldn’t) that is the last step in the workflow (the last should be “read” (GET) from storage, but that step is not yet in the docs).

Looks like an interesting and useful project. I look forward to next release when the docs and containers get in sync and this bug hopefully fixed.

Wish list:

  • More info about max size of blobs that can be PUT
  • GET steps
  • (Maybe this is a stupid question): I wonder if there will be a way to differentiate public vs. private access
  • (May be already documented, didn’t see it): recommended approach to encrypt objects, in the case public access be the default (I can’t tell if it will or won’t, since GET step isn’t in the docs right now)
  • (May be common knowledge for chain storage): it’s not very clear - although it didn’t impact my evaluation - if it’s OK to submit and sign a proposal beforestart_block as long as it’s not published before it. I wouldn’t expect that PUT can be performed before start_block, but maybe other steps can although, even in that case, the question is how much before start_block (i.e. N-14400, N-256, etc.)

Example 1

  • This confusing error in the last step (publish-deal) doesn’t tell you anything. Maybe your balance isn’t enough, maybe your JSON is populated with wrong data, maybe it’s something else… Go figure… This is being fixed, but not fixed yet as far as I can tell. The docs don’t say how much balance is required (there’s a per-block cost formula, but I can’t be sure if there are other costs and in any case the error seems WASM-related and not balance related
Error: RpcClient(Call(ErrorObject { code: InternalError, message: "proposal has not been accepted", data: None }))
  • Server-side log doesn’t tell me much either, and currently there are no open issues related to this.

2024-11-24 01:56:06.020  INFO tokio-runtime-worker runtime::market: [Parachain] on_finalize: no deals to process in block: 168    
2024-11-24 01:56:06.020 DEBUG tokio-runtime-worker runtime::system: [Parachain] [168] 2 extrinsics, length: 3670 (normal 0%, op: 0%, mandatory 0%) / normal weight:Weight(ref_time: 0, proof_size: 0) (0%) op weight Weight(ref_time: 0, proof_size: 0) (0%) / mandatory weight Weight(ref_time: 10532249000, proof_size: 77889) (0%)    
2024-11-24 01:56:06.021  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🎁 Prepared block for proposing at 168 (2 ms) [hash: 0x12621b07d07bb160888e9e729154479000e946168062ee2eff792157a9d7bcee; parent_hash: 0xcc6d…bda4; extrinsics (2): [0x5a3a…8d1c, 0x662d…a456]    
2024-11-24 01:56:06.022  INFO tokio-runtime-worker aura::cumulus: [Parachain] 🔖 Pre-sealed block for proposal at 168. Hash now 0xe6510c8b77d52dea373b714603e2ef6947a13e7d9ba5a3db47906b8d406a740f, previously 0x12621b07d07bb160888e9e729154479000e946168062ee2eff792157a9d7bcee.
2024-11-24 01:56:06.022  INFO tokio-runtime-worker substrate: [Parachain] 🏆 Imported #168 (0xcc6d…bda4 → 0xe651…740f)    
2024-11-24 01:56:06.023  INFO tokio-runtime-worker aura::cumulus: [Parachain] PoV size { header: 0.21875kb, extrinsics: 3.5849609375kb, storage_proof: 8.9638671875kb }
2024-11-24 01:56:06.023  INFO tokio-runtime-worker aura::cumulus: [Parachain] Compressed PoV size: 9.857421875kb
2024-11-24 01:56:06.023 DEBUG tokio-runtime-worker parachain::collation-generation: [Relaychain] candidate is generated candidate_hash=0x3f66be629f2094c229722a38888c5239cbf1fda880835da4557524e8039fe65d pov_hash=0x81c5baeced98def98965cc0d1fc164c0de82bb2a3abcce14023cff9ab72416e9 relay_parent=0xc83f1e6c2f8e391c95c4e3ad97dea52841c28649fe1d6c75673470423d413ede para_id=1000 traceID=84274839485452377306051434561466552889
2024-11-24 01:56:06.024 DEBUG tokio-runtime-worker parachain::collator-protocol: [Relaychain] Received session info session_index=19
2024-11-24 01:56:06.024 DEBUG tokio-runtime-worker parachain::collator-protocol: [Relaychain] Accepted collation, connecting to validators. para_id=1000 candidate_relay_parent=0xc83f…3ede relay_parent_mode=Enabled { max_candidate_depth: 3, allowed_ancestry_len: 2 } candidate_hash=0x3f66be629f2094c229722a38888c5239cbf1fda880835da4557524e8039fe65d pov_hash=0x81c5baeced98def98965cc0d1fc164c0de82bb2a3abcce14023cff9ab72416e9 core=CoreIndex(0) current_validators=[Public(d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d (5GrwvaEF...))] traceID=84274839485452377306051434561466552889
2024-11-24 01:56:06.024 DEBUG tokio-runtime-worker parachain::collator-protocol: [Relaychain] Advertising collation. relay_parent=0xc83f1e6c2f8e391c95c4e3ad97dea52841c28649fe1d6c75673470423d413ede candidate_hash=0x3f66be629f2094c229722a38888c5239cbf1fda880835da4557524e8039fe65d peer_id=12D3KooWQCkBm1BYtkHpocxCwMgR8yjitEeHGx8spzcDLGt2gkBm traceID=84274839485452377306051434561466552889
2024-11-24 01:56:06.024 DEBUG tokio-runtime-worker parachain::validator-discovery: [Relaychain] New ConnectToValidators request peer_set=Collation requested=1 failed_to_resolve=0
2024-11-24 01:56:06.024 DEBUG tokio-runtime-worker parachain::validator-discovery: [Relaychain] New ConnectToValidators resolved request peer_set=Collation num_peers=1 removed=0
2024-11-24 01:56:06.025  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🙌 Starting consensus session on top of parent 0xe6510c8b77d52dea373b714603e2ef6947a13e7d9ba5a3db47906b8d406a740f (#168)    
2024-11-24 01:56:06.025 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/user/.cargo/git/checkouts/polkadot-sdk-cff69157b985ed76/dba2dd5/substrate/frame/aura/src/lib.rs:135:21:
Slot must increase    
2024-11-24 01:56:06.025 ERROR tokio-runtime-worker aura::cumulus: [Parachain] err=Error { inner: Proposing

Caused by:
    0: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x43a5e6 - polka_storage_runtime.wasm!rust_begin_unwind
           1: 0x3fa02 - polka_storage_runtime.wasm!core::panicking::panic_fmt::h58961fa2eb0f3a16
           2: 0x3874b8 - polka_storage_runtime.wasm!<pallet_aura::pallet::Pallet<T> as frame_support::traits::hooks::OnInitialize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_initialize::h79eda421d9430a77
           3: 0x12b384 - polka_storage_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20) as frame_support::traits::hooks::OnInitialize<BlockNumber>>::on_initialize::h6d131c8a66e07cd7
           4: 0x2edafa - polka_storage_runtime.wasm!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::initialize_block::h7cd8d6f9864c02a6
           5: 0x22360e - polka_storage_runtime.wasm!Core_initialize_block
    1: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
       WASM backtrace:
       error while executing at wasm backtrace:
           0: 0x43a5e6 - polka_storage_runtime.wasm!rust_begin_unwind
           1: 0x3fa02 - polka_storage_runtime.wasm!core::panicking::panic_fmt::h58961fa2eb0f3a16
           2: 0x3874b8 - polka_storage_runtime.wasm!<pallet_aura::pallet::Pallet<T> as frame_support::traits::hooks::OnInitialize<<<<T as frame_system::pallet::Config>::Block as sp_runtime::traits::HeaderProvider>::HeaderT as sp_runtime::traits::Header>::Number>>::on_initialize::h79eda421d9430a77
           3: 0x12b384 - polka_storage_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20) as frame_support::traits::hooks::OnInitialize<BlockNumber>>::on_initialize::h6d131c8a66e07cd7
           4: 0x2edafa - polka_storage_runtime.wasm!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::initialize_block::h7cd8d6f9864c02a6
           5: 0x22360e - polka_storage_runtime.wasm!Core_initialize_block

Stack backtrace:
   0: anyhow::error::<impl anyhow::Error>::new
   1: <cumulus_client_consensus_proposer::Proposer<B,T> as cumulus_client_consensus_proposer::ProposerInterface<B>>::propose::{{closure}}
   2: cumulus_client_consensus_aura::collators::lookahead::run_with_export::{{closure}}
   3: <futures_util::future::future::map::Map<Fut,F> as core::future::future::Future>::poll
   4: <sc_service::task_manager::prometheus_future::PrometheusFuture<T> as core::future::future::Future>::poll
   5: <futures_util::future::select::Select<A,B> as core::future::future::Future>::poll
   6: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
   7: tokio::runtime::task::core::Core<T,S>::poll
   8: tokio::runtime::task::harness::Harness<T,S>::poll
   9: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  10: tokio::runtime::scheduler::multi_thread::worker::Context::run
  11: tokio::runtime::context::runtime::enter_runtime
  12: tokio::runtime::scheduler::multi_thread::worker::run
  13: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
  14: tokio::runtime::task::core::Core<T,S>::poll
  15: tokio::runtime::task::harness::Harness<T,S>::poll
  16: tokio::runtime::blocking::pool::Inner::run
  17: std::sys::backtrace::__rust_begin_short_backtrace
  18: core::ops::function::FnOnce::call_once{{vtable.shim}}
  19: std::sys::pal::unix::thread::Thread::new::thread_start
  20: start_thread
             at ./nptl/pthread_create.c:442:8
  21: __GI___clone3
             at ./misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 }
2024-11-24 01:56:06.749  INFO tokio-runtime-worker substrate: [Relaychain] 💤 Idle (2 peers), best: #196 (0xc83f…3ede), finalized #193 (0xe4cd…2da7), ⬇ 1.1kiB/s ⬆ 2.6kiB/s    
2024-11-24 01:56:06.804  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (0 peers), best: #168 (0xe651…740f), finalized #163 (0x755b…760e), ⬇ 72 B/s ⬆ 40 B/s    
2024-11-24 01:56:10.015 DEBUG tokio-runtime-worker parachain::validator-discovery: [Relaychain] New ConnectToValidators request peer_set=Collation requested=1 failed_to_resolve=0
2024-11-24 01:56:10.015 DEBUG tokio-runtime-worker parachain::validator-discovery: [Relaychain] New ConnectToValidators resolved request peer_set=Collation num_peers=1 removed=0
2 Likes
45

Hey @somedude!

Thank you for your through feedback!

We’ve rebuilt the binaries and reviewed the docs to ensure they’re in-sync (like adding the faucet). We also added more information to the book which should answer your questions — in short

  • we don’t support GET yet — we’re working on adding retrieval to the system;
  • the maximum PUT size is 2048 bytes — we’re also working on it,
  • all accesses are public and as such we recommend you encrypt your files (this can be done using 7zip, age, or VeraCrypt, we don’t endorse any in specific),
  • you can and should submit your deals early so that the storage provider has time to process them (implementation details here will vary as work progresses)

Finally, we’d like to ask you to elaborate what was not clear in the install/setup workflow so that we can address that as soon as possible!

In the meantime, we invite you to take a look again and once more, thank you for your feedback!

3 Likes
46

2024-11-26 Update

We’re currently working on:

  • Finishing our spring cleaning — i.e. reducing tech debt
  • PoSt proofs
  • Gregor — the deal aggregator

We’re starting work on:

  • Indexer research
  • Fetching research
1 Like
47

Thanks for taking care of these things and the info is very helpful.

Regarding the maximum size I wondered if something like multipart upload was possible using individual 2KiB chunks, but it seems that needs to be done on the client (and that’s fine) for those who need it. Then maybe use the first chunk to store a map of other chunks, and the rest to store object content. Or maybe put a pointer to the next part at the end of each part that has the next part…

I happened to need just 1KiB, so I’m good for now. Those who need more would probably prefer that one common and tested method of chunking be implemented by the project so that they don’t have to build their own.

The workflow was a bit confusing in terms of the sequence of steps required to get everything deployed. IMO there should be 1 section for deploy/install with 2 subsections (docker, build from source or download + run from CLI), 1 section to getting faucet funds and making sure all the required accounts are there and working, and 1 section to getting things done (regardless of how you run, from Docker or CLI).

I suppose it’s not necessary for people used to working with parachains, but as the first time user I struggled a bit even though I often use a Substrate chain (w/o parachains).

I assume the bug I was encountering was fixed as well? As that was the last step and everything else appears to be working, that’s the only fix I strictly need.

Do you mean this “in general” or related to the bug I encountered?
Regarding the bug I encountered, I would assume that publish-deal should be able to start after the start block (as I’ve committed to paying anyway) and that your comment was generic - in which case I wonder if the docs could define “early” (that would be useful so that we ca add current_height+X where X is the recommended “advance notification”.

Thanks for making these improvements!

48

That’s a great point and chunking is the solution if the allowed size is X and you need to store the file > X. We didn’t plan providing a chunking solution, as the small sector size is experimental - for small file sizes, proving time is fast, so testing time is fast.
We are going to introduce bigger file size limits, probably the ‘unit of storage’ will be 32GiB/64 GiB on-chain.

Files in our chain are stored in sectors (unit of storage), so a Storage Provider gathers deals from the clients and packs them into a sector. Currently the Sector Size is 2KiB, hence the limitation.

When we move to bigger sector sizes, there will be another problem to handle - not chunking but batching. Storage Providers will prefer to store bigger data deals, so if you would like to store small pieces of data (1KiB in 32GiB) you may not find a Storage Provider that is willing to accept a deal. That’s why we’re building a deal aggregation component - ‘Gregor’, it’ll gather small pieces of data from the clients and propose it in a bigger batch.

I hope I understood your concerns correctly and explained our reasoning, if not, looking for follow-up questions and we’re here!

1 Like
49

Thanks for the update, yes, that’s the right way.

I like Gregor - “Gregor The Aggregator” :smiley:

There are “traditional” (lol) Web3 players that heavily favor big blobs, so I think there’s space for these tiny ones (if you need just a few) and medium-sized ones (say up to 32MB, enough for many apps).

Above that is good and fine, but also overlaps with those “bulk” storage options and may be much pricier per MB than bulk storage. Still, it may be desirable for folks who need a few GB and prefer to use 1 provider/back-end for many small & medium plus some large, than code and test two integrations and pay to multiple providers (complexity, cost, inconvenience).

50

Currently, publish_deals should always be done before the start_block, any deal published after the start_block will be rejected by the chain. There is no recommended advance notice, as long as the deals are submitted before the start_block they will be accepted.
However, if it’s done too late, then there won’t be enough time to do pre-commit and prove the sectors, we’ll figure out appropriate values and provide them as a recommendation (we haven’t estimated them yet) - thank you for noticing that!

Regarding the documentation, we will review them and improve on the clarity and flow of the documentation.

Thank you for your extensive feedback and helping us improve :pray:.

1 Like
51

Whoa - that’s interesting! Didn’t see that coming. I’ve tried both before and after, IIRC, but I’ll avoid “after” from now on. It’d be good to have these points highlighted in the docs.

Hit a small landmine with binaries this time (Ubuntu 22.04 LTS has an older glibc) so I’m building binaries instead. Sys reqs are technically correct (e.g. if you run from the source or use Docker, but not if you use the prebuilt binaries) which is an example of why I suggested to structure the docs in distinct stages. Not a big problem now that I understand this area better - just FYI.

Another concrete example is that build from source doesn’t mention (so this is an exercise for the reader) all binaries required to run - some (external) binaries need to be downloaded even if you build from the source. Makes sense once you compare the pages, but not until then :slight_smile:

Thanks for the info!

│ Command                 │ bash -c polka-storage-node build-spec  --disable-default-bootnode > /tmp/zombie-dc56376f7f0702cc4… │
└─────────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────┘
Error:  	 Error: Command failed with exit code 1: bash -c polka-storage-node build-spec  --disable-default-bootnode > /tmp/zombie-dc56376f7f0702cc4e627013b6c9d277_-3343252-7vTh1SAP4wOs/cfg/undefined-plain.json
polka-storage-node: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.38' not found (required by polka-storage-node)
polka-storage-node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.39' not found (required by polka-storage-node)
polka-storage-node: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.38' not found (required by polka-storage-node)
1 Like
52

2024-12-03 Update

We’ve worked on improving the docs.

We’re working on:

  • Researching randomness methods
  • Researching the indexer node
  • Researching fetching methods
  • Implementing PoSt in the pipeline
  • Researching for the Gregor POC
1 Like
53

Regarding research on randomness methods, this might be of interest: Ideal Labs | Updates & Next Steps - #3 by driemworks

We’re working towards introducing publicly verifiable randomness-as-a-service to Polkadot.

4 Likes
54

2024-12-10 Update

:man_scientist: We’re currently researching solutions for:

We’re pushing forward our proving implementation, we’ve merged ongoing PoSt submissions and we’re working on fixing related bugs and supporting proving multiple partitions.

We’re still working on the Gregor POC.

Finally, we’ve open sourced our Delia POC :rocket: — GitHub - eigerco/delia: Web application for proposing and publishing deals with providers on the Polka Storage System Parachain.

3 Likes
55

2024-12-17 Update

:christmas_tree: We have merged:

Ongoing:

56

2024-12-24 Update

We’ve solved an handful of issues:

We’ve merged the randomness weights:

And progress continues on retrieval and Gregor.

:christmas_tree: Happy Holidays!

1 Like
57

2025-01-08 Update

Progress continues on benchmarks:

And on the proving pipeline:

As well as on extending our CAR library:

And some work into the P2P aspects of the Storage Provider:

58

2025-01-14 Update

We’ve added configuration file support:

We’re still working on retrieval and P2P