Polkadot Native Storage Updates

Hello everyone,

I’m Serhii from the Eiger team. As a follow-up to our efforts in building native storage, presented in this thread, we would like to provide weekly updates in this thread regarding the progress we are making.

Here is the current status from the scope of our proposal as of March 25, 2024:

Milestone: 1. Research.

  • Task: 1. Research lotus.
    • Subtasks:
        1. Analysis of lotus.
        • Prepared an overview of Filecoin specification and Lotus implementation analysis.
        • Prepared parachain implementation guidelines with project structure definition, required pallets for implementation. Also, testing setup and instructions have been provided.
        1. Define JSON-RPC API.
        • WIP: Defined and summarized the RPC API provided by Lotus with partial cross-references to required methods for implementation in our parachain solution.
        • WIP: Partially defined RPC methods to be exposed and described methods which are already exposed by Substrate.
  • Task: 2. Research lotus-miner.
    • Subtasks:
        1. Analysis of lotus-miner.
        • WIP: Conducted lotus-miner analysis and prepared an overview of the component.
        1. Define JSON-RPC API.
        • WIP: Defined and summarized the RPC API provided by Lotus, with cross-references to the required methods for implementation in our parachain solution.
        1. Analyze the lotus-worker.
        • WIP: Conducting a lotus-worker analysis to decide whether we should implement it and how.
        • WIP: Overview of RPC method definitions.

:bulb: The next update reports will be provided on a weekly basis and will only include and describe progress made since the previous report.

2 Likes

We should observe that storage depends upon threat model, which blockchain storage proposals typically either omit entirely or else over engeneering, at extreme costs.

“Availability” systems incur real costs, with the one used by polkadot being optimal for polkadot’s byzantine threat model. Celesia costs more because they over sample. Filecoin & ETH cost more because they prove encoding roots using zk proofs.

Afaik nobody has proposed a storage scheme with a formalized threat model that provides less assurances than polakdot’s availability, but still provides useful assurances, except maybe certificate transperency. I suspect the answer lies with rateless erasure codes, and assume an honest encoder, but the availabiity work using rateless erasure codes winds up much worse than even celestia.

I always phrase this question as: How could rateless erasure codes improve bittorrent?

Anyways…

I relaize this project ports filecoin, which likely serves some purposes, but it’ll be interesting to see how much more this costs than our own blockspace.

1 Like

Thanks for sharing your thoughts on the challenges and costs involved in blockchain storage. We’re really excited about where we’re heading with our project.

When planning the project, we took inspiration from Lotus, adopting their well-tested decentralized storage protocols and algorithms, but we’re tweaking them to better serve Polkadot’s unique needs. This ensures our solution isn’t just a copy, but a tailored fit for our ecosystem.

On pricing, we’re looking to Filecoin’s model for inspiration. We plan to let market forces determine our storage prices. That way we foster a competitive atmosphere that promotes efficiency and fairness, encouraging providers to offer the best, most cost-effective services possible, meeting the diverse needs of our community.

2 Likes

2024-04-05 Weekly Update

Milestone: 1. Research.

  • Task: 1. Research lotus.
    • Subtasks:
        1. Analysis of lotus.
        • Marked as done.
        1. Define JSON-RPC API
        • Defined and summarized the RPC API provided by Lotus with cross-references to required methods for implementation in our parachain solution.
        • Defined RPC methods to be exposed by parachain node.
        • Described methods which are already exposed by Substrate.
        1. Research markets and boost
        • WIP: Research of finite state machines (FSM), markets and boost components.
  • Task: 2. Research lotus-miner.
    • Subtasks:
        1. Analysis of lotus-miner
        • Conducted lotus-miner analysis and prepared an overview of the component.
        1. Define JSON-RPC API
        • WIP: Defined and summarized the RPC API provided by Lotus, with cross-references to the required methods for implementation in our parachain solution.
        1. Analyze the lotus-worker
        • WIP: Conducting a lotus-worker analysis to decide whether we should implement it and how.
        • WIP: Overview of RPC method definitions.
1 Like

2024-04-13 Weekly Update

Milestone: 1. Research.

  • Task: 1. Research lotus.
    • Subtasks:
        1. Define JSON-RPC API
        • Finalized the RPC API, and the subtask is marked as done.
        1. Research markets and boost
        • WIP: Research of finite state machines (FSM), markets and boost components.
  • Task: 2. Research lotus-miner.
    • Subtasks:
        1. Analysis of lotus-miner
        • Marked as done.
        1. Define JSON-RPC API
        • WIP: Defined and summarized the RPC API provided by Lotus, with cross-references to the required methods for implementation in our parachain solution.
        1. Analyze the lotus-worker
        • WIP: Conducting a lotus-worker analysis to decide whether we should implement it and how.
        • WIP: Overview of RPC method definitions.
1 Like

2024-04-22 Weekly Update

Milestone: 1. Research.

  • Task: 1. Research lotus.
    • Subtasks:
        1. Research markets and boost.
        • Extended documentation to include research on finite state machines (FSM) results, markets, and boost components. The subtask is marked as done.
  • Task: 2. Research lotus-miner.
    • Subtasks:
        1. Define JSON-RPC API.
        • WIP: Defined and summarized the RPC API provided by Lotus, with cross-references to the required methods for implementation in our parachain solution.
        1. Analyze the lotus-worker.
        • WIP: Conducting a lotus-worker analysis to decide whether we should implement it and how.
        • WIP: Overview of RPC method definitions.
  • Task: 3. Research FVM.
    • Subtasks:
        1. Research compiling the FVM for WASM.
        • WIP: Conducting research on compiling the FVM for WASM.
        1. Specification of Actor messages.
        • WIP: Definition of actors overview with specifications. Definition of the API required for implementation.

2024-05-02 Weekly Update

Milestone: 1. Research.

  • Task: 2. Research lotus-miner.
    • Subtasks:
        1. Define JSON-RPC API.
        • WIP: Defined and summarized the RPC API provided by Lotus, with cross-references to the required methods for implementation in our parachain solution.
        1. Analyze the lotus-worker.
        • WIP: Conducting a lotus-worker analysis to decide whether we should implement it and how.
        • WIP: Overview of RPC methods definitions.
  • Task: 3. Research FVM.
    • Subtasks:
        1. Research compiling the FVM for WASM.
        • WIP: Conducting research on compiling the FVM for WASM.
        1. Specification of Actor messages.
        • WIP: Definition of actors overview with specifications. Definition of the API required for implementation.
1 Like

2024-05-11 Weekly Update

Milestone: 1. Research.

  • Task: 2. Research lotus-miner. All subtasks of this task have been resolved and marked as completed.
    • Subtasks:
        1. Define JSON-RPC API.
        • The RPC API provided by Lotus has been defined, finalized, and marked as done.
        1. Analyze the lotus-worker.
        • Conducted and marked as done a Lotus Worker analysis to decide whether we should implement it and how.
        • Conducted an overview of RPC method definitions.
  • Task: 3. Research FVM.
    • Subtasks:
        1. Research compiling the FVM for WASM.
        • WIP: Conducting research on compiling the FVM for WASM.
        1. Specification of Actor messages.
        • WIP: Definition of actors overview with specifications. Definition of the API required for implementation.

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • Started work on porting actors’ functionality to the parachain based on the guidelines provided in the process of the first milestone implementation.

Milestone: 3. polka-store.

  • Task: 9. External Libraries.
    • Started work on low-level storage implementation based on the guidelines provided in the process of the first milestone implementation.
1 Like

2024-05-19 Weekly Update

Milestone: 1. Research. All tasks and subtasks of this milestone have been resolved and marked as completed.

  • Done: Task: 3. Research FVM.
    • Subtasks:
        1. Done: Research compiling the FVM for WASM.
        • Done: Conducted research on compiling the FVM for WASM.
        1. Specification of Actor messages.
        • Done: Prepared actors overview with specifications. Definition of the API required for implementation.

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided in the process of the first milestone implementation.

Milestone: 3. polka-store.

  • Task: 9. External Libraries.
    • WIP: Low-level storage implementation based on the guidelines provided in the process of the first milestone implementation.
2 Likes

2024-05-28 Weekly Update

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided in the process of the first milestone implementation.
      • WIP: Market Pallet Implementation.
      • WIP: Storage Provider Pallet implementation.
      • WIP: Collator Power Pallet Implementation.

Milestone: 3. polka-store.

  • Task: 9. External Libraries.
    • WIP: Low-level storage implementation based on the guidelines provided in the process of the first milestone implementation.
      • WIP: Rust CARv2 implementation.
      • WIP: Indexing mechanism implementation.
2 Likes

2024-06-03 Weekly Update

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided in the process of the first milestone implementation.
      • WIP: Market Pallet Implementation.
      • WIP: Storage Provider Pallet implementation.
      • WIP: Collator Power Pallet Implementation.

Milestone: 3. polka-store.

  • Task: 9. External Libraries.
    • WIP: Low-level storage implementation based on the guidelines provided in the process of the first milestone implementation.
      • Done: Rust CARv2 implementation.
      • WIP: Indexing mechanism implementation.
      • WIP: Implementation of the polka-storage-provider component, which serves as the entry point from the Storage Provider’s perspective.
3 Likes

2024-06-13 Weekly Update

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided in the process of the first milestone implementation.
      • WIP: Market Pallet Implementation.
      • WIP: Storage Provider Pallet implementation.

Milestone: 3. polka-store.

  • Task: 9. External Libraries.
    • WIP: Low-level storage implementation based on the guidelines provided in the process of the first milestone implementation.
      • WIP: Indexing mechanism implementation.
      • WIP: Implementation of the “polka-storage-provider” component, which serves as the entry point from the Storage Provider’s perspective:
        • Done: Subkey commands integration into the entry point crate.
        • Done: RPC server functionality integration.
2 Likes

2024-06-22 Weekly Update

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided in the process of the first milestone implementation.
      • WIP: Payment Channels Pallet Implementation.
      • WIP: Market Pallet Implementation.
      • WIP: Storage Provider Pallet implementation.

Milestone: 3. polka-store.

  • Task: 9. External Libraries.
    • WIP: Low-level storage implementation based on the guidelines provided in the process of the first milestone implementation.
      • Done: Implementation of the storage indexing mechanism.
      • WIP: Implementation of the polka-storage-provider component, which serves as the entry point from the Storage Provider’s perspective.
      • Done: Implementing benchmarks for the library to read and write CAR files.
2 Likes

2024-07-08 Update

General:

WIP: Testing the Phase 1 deliverables.
WIP: Preparing a demonstration of the Phase 1 deliverables.

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided during the first milestone implementation.
      • WIP: Storage provider pallet implementation.
      • Done: Market Pallet logic: terminate_sectors, verify_deals_for_activation, activate_deals, cron_tick (hook for detecting expired sectors).
      • Done: Market Pallet: Extrinsics: publish_storage_deals settle_deal_payments,add_balance, withdraw_balance, on_terminate_sector.
      • Done: implemented: settle_deal_payments
      • Done: add_balance, verify_deals_for_activation, activate_deals, cron_tick.

Milestone: 3. polka-store.

  • Task: 9. External Libraries.
    • Done: RPC bootstrap for polka-storage-provider component.
    • Finished: Low-level storage CARv2 implementation based on the guidelines provided during the first milestone implementation.
1 Like

General:

Ongoing: Testing the Phase 1 deliverables.

  • Integration tests development

WIP: Preparing a demonstration of the Phase 1 deliverables.

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided during the first milestone implementation.
      • WIP: Storage provider pallet implementation.
        • Ongoing: terminate_sector, submit_windowed_post, declare_faults and declare_faults_recoveredextrinsic.

Milestone: 3. polka-store.

  • Task: 9. External Libraries — Done
2 Likes

2024-07-23 Update

Ongoing: Testing the Phase 1 deliverables.

  • Integration tests development
  • Bug fixing as we find them
  • Implementing a CLI for interaction with extrinsics and for integration testing

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided during the first milestone implementation.
      • WIP: Storage provider pallet implementation.
        • Ongoing: terminate_sector, submit_windowed_post, declare_faults and declare_faults_recoveredextrinsic.
2 Likes

2024-07-31 Update

What has the team done since last week?

Ongoing: Testing the Phase 1 deliverables.

  • Integration tests development
  • Bug fixing as we find them
  • Base CLI was implemented, currently extending it with more extrinsics

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided during the first milestone implementation.
      • Done: submit_windowed_post
      • WIP: Storage provider pallet implementation.
        • Ongoing: terminate_sector, declare_faults and declare_faults_recoveredextrinsic.
1 Like

2024-08-06
What has the team done since last week?

Ongoing: Testing the Phase 1 deliverables.

  • Integration tests development
  • Bug fixing as we find them
  • Done: added the submit_windowed_post to the extrinsics CLI

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided during the first milestone implementation.
      • Done: pre-commit hook mechanism
      • WIP: Storage provider pallet implementation.
        • Ongoing: terminate_sector, declare_faults and declare_faults_recoveredextrinsic.

What has the team done since last week?

Ongoing: Testing the Phase 1 deliverables.

  • Integration tests development
  • Bug fixing as we find them

Milestone: 2. Collator Node.

  • Task: 4. 2.1 Port Actors to pallets.
    • WIP: Porting actors’ functionality to the parachain based on the guidelines provided during the first milestone implementation.
      • Done: Storage provider pallet implementation.
        • declare_faults and declare_faults_recovered extrinsic.
      • Ongoing:
        • Testing & bug fixing
        • terminate_sector
1 Like

2024-08-20

What has the team done since last week?

Ongoing: Testing the Phase 1 deliverables.

  • Integration tests development
  • Bug fixing as we find them
  • Writing end-user documentation