In other words, we (as in the blockchain ecosystem in general) haven’t found a good way to store data “somewhere” and being sure to be able to retrieve it later. IPFS for example doesn’t guarantee that you’ll retrieve it later, and Filecoin last time I’ve checked doesn’t really work and has oriented itself towards cold storage anyway.
Don’t pinning services guarantee that you can retrieve IPFS data later?
I have been using a pinning service for a couple months now to ensure that my IPFS content stays available.
A recent issue I encountered is IPNS reliability; my website/pages were unavailable for a couple days because of a 504 Gateway Time-Out. As a quick-fix I redirected my primary domain to the pinning gateway server.
I am curious, how are others in the ecosystem handling data storage and availability?
If the owner of a pinning service decided to shut it down, everyone would lose their data without any automatic fallback.
And just like every centralized service, they work by reputation. A pinning service could simply decide to throw away your data if they wanted. You have no guarantee that they won’t delete your data, you just have their word that they won’t do it.
Because they work by reputation, everyone ends up using the same pinning service (not trusting small alternatives), which increases the power of the owners of the service, etc. All the drawbacks of centralization.
They are free not because they don’t have monthly costs but because they’re subsidized (I don’t really know by who, maybe Protocol Labs, or a treasury, or they have a premium service). Every file that you pin increases their monthly bill. I suspect that if the volume of data was high, it wouldn’t be free anymore.
Actually now that you point it out (the centralized nature), I realize that the Crust Network explainer video also glosses over the centralized Web3 IPFS Gateway and Web3 IPFS Pinner. Hopefully it gets to enough incentivized participants where centralized pinning won’t be needed.
Are there any active teams or organizations currently addressing the challenge of decentralized data storage and availability?
Exploring alternative approaches to data storage and retrieval, especially those involving the ecosystem’s resources, is valuable to envisioning a decentralized and resilient future.
I hold the conviction that mere protocol adjustments alone will not resolve the decentralized data availability predicament. It necessitates a comprehensive reimagining of computing methodologies and storage accessibility.
I appreciate the work being done by IPFS and Filecoin, but their limitations of data availability, network scalability, content discovery, file persistence, incentivization, file integrity, and centralization risks are all challenges that I see being best handled from a zeroth-principled perspective. Creating a novel approach towards specialized hardware devices designed specifically for decentralized storage can provide a dedicated and reliable means of storing data.
Which has me wondering, if hardware is the solution, then why haven’t we done it yet?
Well, the absence of widespread alternatives to IPFS are attributed to the technical challenges, resource limitations (financial, manufacturing, etc.), adoption challenges (incentivization), trade-offs/imbalances for user experiences, and the simple fact that software solutions are always preferred over hardware alternatives. It would take time and money to build.
I disagree.
This problem is a problem anyone can solve with no money.
The problem is easy to understand in 5 minutes. I’ve explained it in this post. All you need to try solve it is your brain and optionally a sheet of paper. If you find a good way to solve it, other people will build it.
We’ve had 5 years of high market cap where tens of billions flooded the blockchain ecosystem. If this proved anything, it’s that incentivizations and money alone don’t solve problems.
I respectfully disagree with the notion that the problem of decentralized storage can be solved without any financial investment.
The issue at hand is multi-dimensional and requires a comprehensive approach to address various facets effectively.
The Problem:
The prisoner’s dilemma that arises in decentralized storage systems. In such systems, where data is stored and accessed by multiple participants, there is no strong individual incentive for users to store and provide data to others. This lack of incentive can lead to a situation where only a few participants store the data, causing an imbalance and potential disruption of the service. While goodwill initially motivates some individuals to contribute, it becomes challenging to sustain decentralized storage for large-scale applications like Facebook due to the massive volume of data generated. There is also difficulty in designing effective solutions, such as treasury systems, to incentivize participants without the risk of cheating or free-riding.
Physical Storage Device:
The introduction of a device that engages the average user, is consumer-friendly, is easy to use, and seamlessly integrates with existing infrastructure would create a strong monetary incentive for users to participate in decentralized storage networks, enabling them to earn passive income through data storage activities (It’s a salt lamp that makes them money).
It provides individuals with a tangible and visible representation of their involvement in Web3 which further cements their commitment to the network.
Plug & play functionality - Attracts better actors who will simply want to plug it in and forget about it, happy to collect their rewards.
Stake - the consumer bought a physical device with a specialized purpose.
Monetary incentive - this device is correlated with reward generation.
Web3 income - the generated income from selling these devices can be reinvested into further research and development, self-assemblying systems, or development promoting innovation and growth within the Web3 ecosystem.
Cultural change - drives the shift needed to bring Web3 techologies into fruition, altering the way people engage with the Web.
Data Economy:
Web3 is driving the transition towards a data economy by embracing decentralized technologies as its backbone. The Web3 ethos is a set of principles that aim to empower individuals, promote user ownership and control over data, and foster a more open and inclusive digital ecosystem. It requires decentralized storage in order for data to make Web3 fruitful and hence is the main economy (energy) pushing Web2 into Web3.
Incentivization Mechanisms - while it is true that some individuals may be driven by ideological, philosophical, or technical motivations to participate in decentralized storage networks, it is essential to recognize that the majority of users require additional incentives and easier access points. The physical device and introduction of the data economy is what gets more users to become engaged with the network.
User Empowerment - granting individuals control over their data and privacy is a cornerstone of the data economy. By leveraging decentralized storage, users can take ownership of their personal information, decide who has access to it, and retain the ability to monetize or share it selectively. This paradigm shift empowers individuals to actively engage in the digital landscape and become stakeholders in the value generated from their data. Decentralized storage then doesn’t rely on good will, but is seen as a sustainable and valuable part of engaging with the web.
Value Creation & Redistribution - value creation is no longer monopolized by centralized entities. The redistribution of value occurs as users actively participate, share their data, and contribute to the network. It’s the bottom up approach. It enables users from diverse backgrounds to engage in economic activities, earn income, and build sustainable livelihoods by leveraging their data assets within the data economy.
If we say decentralized storage is valuable, then actually make it valuable and fully hold it as the cornerstone of Web3’s market. Data will then be able to fruitfully fill the landscape and lots of innovation, R&D, etc. becomes possible because now the data is available for the public to thrive on and not just monopolostic corporate giants.
Cheating and Fraud:
You implement robust measures and protocols.
Verification & Auditing
Reputation Systems
Slashing Conditions
Strong mechanisms & protocols
Monitoring & detection
Education & awareness
In summary:
All these components collectively form a comprehensive framework necessary to incentivize participation, overcome the prisoner’s dilemma, and foster the growth of decentralized storage networks.
This holistic approach establishes decentralized storage as the backbone of Web3, fostering a more open and inclusive digital ecosystem. Addressing the complexities of decentralized storage requires a multi-dimensional approach that encompasses various elements such as incentives, physical tangibility, usability, cultural change, and economic models. By combining these components and introducing a consumer-friendly storage device, we can foster an environment that attracts the average user, encourages participation, and propels the transition towards a decentralized data economy (which is the sustainable backbone of Web3). Financial investment is essential in this process.
I have nothing against you, but your post is completely off topic to me.
Everybody agrees with what you say, but they are just empty words. The data storage problem is a very pragmatic problem that requires a very pragmatic answer.
Which robust measures and protocols is precisely the problem.
In 6 years of blockchain boom, and despite the fact that this problem has been known from the very start, nobody has figured out how to design these “robust measures and protocols”.
Will continue the thread here, as its a meaty one.
I think the opposite - this is exactly on topic once we start with the following assumption:
When you begin here, and stop trying to introduce unnessary tokens and blockchains and move towards vertical integration of an open-source operating system and hardware ecosystem, dependent on actual revenue models we will make progress.
Decentralised social networking has been solved. Just not with tokens and blockchains.
Sadly, when you begin with this core insight, the vast majority of the current ecosystem is effectively redundant - tokens as products, attempting to reverse their way into products and services.
Sadly people aren’t economically incentivised to face up to that fact, because, guess what, they want NUMBER GO UP! This leads us inexorably into a weird dream state - a beautiful delusion, where large groups of people - founders, investors and contributors aka ‘communities’ convince themselves that they need a token, when they almost certainly don’t.
I have nothing against the core desire for people wanting to better their situations, but the crypto-industry is for the largest part a giant fraud, a multi-level marketing scheme, committed against a naive new investment class - that uses the tools of the attention economy to market ‘Web3’.
People continue to create a better Web using the broken incentives of the attention economy to bootstrap their products and services tokens. This is not possible.
Crypto-currencies such as Bitcoin, Ethereum and Polkadot were created by visionaries who hoped to offer an escape from the crushing gravity of central banks, Wall Street and tech monopolies respectively.
Whilst each engineered solutions that advanced the capabilities of software to establish and transact value in peer to peer networks, their adoption and subsequent narratives were each co-opted and then dictated by a media ecosystem whose core incentives steadily usurped the revolutionary ideals of the founders.
I can also take psychedelics and spit out a lot of ideas, at the end of the day we need something pragmatic. You can invent anything if you take assumptions. For example if I assume that P=NP I can un-hash data. It’s always simple when you say it with words.
Ideas are nearly worthless. If you have a concrete idea, please write down a document explaining the concrete details in the same level of details as what papers usually do.
Pragmatically speaking, there’s no good solution to storing data in a decentralized way right now, and not even a good concept.
Every collective stores its own data and runs its own servers - individually and as a group per the previous points about hardware.
Every collective is a peer / mirror to other collectives via ActivityPub or something like Bison Relay (which uses no accounts, but instead relies on peers opening channels.
Bison Relay is an asynchronous client-server protocol that makes heavy use of the Decred Lightning Network, where every message is sent encrypted, metadata-minimized, and paid for via LN micropayment. The Bison Relay server is accountless and every message is handled individually, where it is both paid for prior to being sent and then received. Bison Relay tightly integrates payments, messaging, and social media, and implements a peer-to-peer social media functionality of making posts to subscribers, subscribing to users’ posts, relaying posts, and replying to posts and comments on posts.
Using Bison Relay requires a small amount of Decred, e.g. 0.1 DCR or less, and this will last many months for regular chatting, but its longevity may vary based on how much data is sent. The billing rate is 1 atom per kilobyte (0.00000001 DCR per kB) to send data and 1 atom per message received.
Typical social media or chat services have a database table of their users, so user A can request to be connected with user B and the server routes the messages. Since server-based routing of messages has been replaced by client-based routing, clients must connect with each other without the use of a server “phonebook”, i.e. lookup table. Clients may connect either via out-of-band invite, which can be sent over other chat or relay networks, preferably via a secure channel, or via Bison Relay directly using mediated key exchanges. This mediated key exchange process occurs roughly as follows:
Bob is in contact with Alice and Carol, but Alice and Carol are not in contact over Bison Relay.
Bob posts something that both Alice and Carol can see.
Alice makes a comment on the post.
Carol sees Alice’s comment on the post.
Carol decides to initiate a mediated key exchange to connect directly with Alice.
Bob mediates the key exchange requested by Carol with Alice.
Carol and Alice are connected directly.
A similar process can occur across several intermediate peers that exist in between the 2 peers that are attempting to connect. Mediated key exchanges avoid the need for an authoritative lookup table by reverting to the “old” meatspace model of meeting people, where one is introduced to new people via people they already know.
Analogous to reposting on various existing social media platforms, clients can “relay” another client’s post to their own followers. If several clients relay a single post in a series, the scenario described above with several intermediate peers existing between 2 peers attempting to connect can occur.
There is a long form research paper coming as you request - it is emerging from this kind of conversation and from active and sustained contribution to these networks.
Its insights, strategies and recommendations are derived in part by this process of debate.
So respectfully I disagree, ideas have enormous power, because they are the foundations of the stories we believe, and when it comes to coordinating humans, belief in stories is at the core of everything.
I agree with this statement from a funding cost/risk perspective…
I know folks who have done iOT hardware/software startups. The successful one was smart door locks. The other one is doing video based fall-detection for aging parents. There are of course many factors at play, but I believe the underlying difficulty is the target market size and the risk of losing investment. With hardware, there is an economy of scale problem.
My reference is anecdotal, but diving into integration hardware solutions right doesn’t seem to make sense. We would probably have to fund somebody to build out the prototype and then network with hardware manufacturers in the hopes that they would integrate the solution for a small addressable market. So very expensive investment, high probability of failure.
Software on top of preexisting hardware is the way to go. With software the only thing you burn is the opportunity cost of time. And if the software isn’t successful someone might be able to repurpose or build upon the intellectual property (code) assets that were built.
Thank you for vociferousness! I recommend you check out EthStorage (video), which while you probably couldn’t implement this forum with it its web3:// , it doesn’t have the “pinning” problem and is a sincere effort.
Can you point out why its not a good concept for us?