In other words, we (as in the blockchain ecosystem in general) haven’t found a good way to store data “somewhere” and being sure to be able to retrieve it later. IPFS for example doesn’t guarantee that you’ll retrieve it later, and Filecoin last time I’ve checked doesn’t really work and has oriented itself towards cold storage anyway.
Don’t pinning services guarantee that you can retrieve IPFS data later?
I have been using a pinning service for a couple months now to ensure that my IPFS content stays available.
A recent issue I encountered is IPNS reliability; my website/pages were unavailable for a couple days because of a 504 Gateway Time-Out. As a quick-fix I redirected my primary domain to the pinning gateway server.
I am curious, how are others in the ecosystem handling data storage and availability?
If the owner of a pinning service decided to shut it down, everyone would lose their data without any automatic fallback.
And just like every centralized service, they work by reputation. A pinning service could simply decide to throw away your data if they wanted. You have no guarantee that they won’t delete your data, you just have their word that they won’t do it.
Because they work by reputation, everyone ends up using the same pinning service (not trusting small alternatives), which increases the power of the owners of the service, etc. All the drawbacks of centralization.
They are free not because they don’t have monthly costs but because they’re subsidized (I don’t really know by who, maybe Protocol Labs, or a treasury, or they have a premium service). Every file that you pin increases their monthly bill. I suspect that if the volume of data was high, it wouldn’t be free anymore.
Actually now that you point it out (the centralized nature), I realize that the Crust Network explainer video also glosses over the centralized Web3 IPFS Gateway and Web3 IPFS Pinner. Hopefully it gets to enough incentivized participants where centralized pinning won’t be needed.
I disagree.
This problem is a problem anyone can solve with no money.
The problem is easy to understand in 5 minutes. I’ve explained it in this post. All you need to try solve it is your brain and optionally a sheet of paper. If you find a good way to solve it, other people will build it.
We’ve had 5 years of high market cap where tens of billions flooded the blockchain ecosystem. If this proved anything, it’s that incentivizations and money alone don’t solve problems.
I have nothing against you, but your post is completely off topic to me.
Everybody agrees with what you say, but they are just empty words. The data storage problem is a very pragmatic problem that requires a very pragmatic answer.
Which robust measures and protocols is precisely the problem.
In 6 years of blockchain boom, and despite the fact that this problem has been known from the very start, nobody has figured out how to design these “robust measures and protocols”.
Will continue the thread here, as its a meaty one.
I think the opposite - this is exactly on topic once we start with the following assumption:
When you begin here, and stop trying to introduce unnessary tokens and blockchains and move towards vertical integration of an open-source operating system and hardware ecosystem, dependent on actual revenue models we will make progress.
Decentralised social networking has been solved. Just not with tokens and blockchains.
Sadly, when you begin with this core insight, the vast majority of the current ecosystem is effectively redundant - tokens as products, attempting to reverse their way into products and services.
Sadly people aren’t economically incentivised to face up to that fact, because, guess what, they want NUMBER GO UP! This leads us inexorably into a weird dream state - a beautiful delusion, where large groups of people - founders, investors and contributors aka ‘communities’ convince themselves that they need a token, when they almost certainly don’t.
I have nothing against the core desire for people wanting to better their situations, but the crypto-industry is for the largest part a giant fraud, a multi-level marketing scheme, committed against a naive new investment class - that uses the tools of the attention economy to market ‘Web3’.
People continue to create a better Web using the broken incentives of the attention economy to bootstrap their products and services tokens. This is not possible.
Crypto-currencies such as Bitcoin, Ethereum and Polkadot were created by visionaries who hoped to offer an escape from the crushing gravity of central banks, Wall Street and tech monopolies respectively.
Whilst each engineered solutions that advanced the capabilities of software to establish and transact value in peer to peer networks, their adoption and subsequent narratives were each co-opted and then dictated by a media ecosystem whose core incentives steadily usurped the revolutionary ideals of the founders.
I can also take psychedelics and spit out a lot of ideas, at the end of the day we need something pragmatic. You can invent anything if you take assumptions. For example if I assume that P=NP I can un-hash data. It’s always simple when you say it with words.
Ideas are nearly worthless. If you have a concrete idea, please write down a document explaining the concrete details in the same level of details as what papers usually do.
Pragmatically speaking, there’s no good solution to storing data in a decentralized way right now, and not even a good concept.
Every collective stores its own data and runs its own servers - individually and as a group per the previous points about hardware.
Every collective is a peer / mirror to other collectives via ActivityPub or something like Bison Relay (which uses no accounts, but instead relies on peers opening channels.
Bison Relay is an asynchronous client-server protocol that makes heavy use of the Decred Lightning Network, where every message is sent encrypted, metadata-minimized, and paid for via LN micropayment. The Bison Relay server is accountless and every message is handled individually, where it is both paid for prior to being sent and then received. Bison Relay tightly integrates payments, messaging, and social media, and implements a peer-to-peer social media functionality of making posts to subscribers, subscribing to users’ posts, relaying posts, and replying to posts and comments on posts.
Using Bison Relay requires a small amount of Decred, e.g. 0.1 DCR or less, and this will last many months for regular chatting, but its longevity may vary based on how much data is sent. The billing rate is 1 atom per kilobyte (0.00000001 DCR per kB) to send data and 1 atom per message received.
Typical social media or chat services have a database table of their users, so user A can request to be connected with user B and the server routes the messages. Since server-based routing of messages has been replaced by client-based routing, clients must connect with each other without the use of a server “phonebook”, i.e. lookup table. Clients may connect either via out-of-band invite, which can be sent over other chat or relay networks, preferably via a secure channel, or via Bison Relay directly using mediated key exchanges. This mediated key exchange process occurs roughly as follows:
Bob is in contact with Alice and Carol, but Alice and Carol are not in contact over Bison Relay.
Bob posts something that both Alice and Carol can see.
Alice makes a comment on the post.
Carol sees Alice’s comment on the post.
Carol decides to initiate a mediated key exchange to connect directly with Alice.
Bob mediates the key exchange requested by Carol with Alice.
Carol and Alice are connected directly.
A similar process can occur across several intermediate peers that exist in between the 2 peers that are attempting to connect. Mediated key exchanges avoid the need for an authoritative lookup table by reverting to the “old” meatspace model of meeting people, where one is introduced to new people via people they already know.
Analogous to reposting on various existing social media platforms, clients can “relay” another client’s post to their own followers. If several clients relay a single post in a series, the scenario described above with several intermediate peers existing between 2 peers attempting to connect can occur.
There is a long form research paper coming as you request - it is emerging from this kind of conversation and from active and sustained contribution to these networks.
Its insights, strategies and recommendations are derived in part by this process of debate.
So respectfully I disagree, ideas have enormous power, because they are the foundations of the stories we believe, and when it comes to coordinating humans, belief in stories is at the core of everything.
I agree with this statement from a funding cost/risk perspective..
I know folks who have done iOT hardware/software startups. The successful one was smart door locks. The other one is doing video based fall-detection for aging parents. There are of course many factors at play, but I believe the underlying difficulty is the target market size and the risk of losing investment. With hardware, there is an economy of scale problem.
My reference is anecdotal, but diving into integration hardware solutions right doesn’t seem to make sense. We would probably have to fund somebody to build out the prototype and then network with hardware manufacturers in the hopes that they would integrate the solution for a small addressable market. So very expensive investment, high probability of failure.
Software on top of preexisting hardware is the way to go. With software the only thing you burn is the opportunity cost of time. And if the software isn’t successful someone might be able to repurpose or build upon the intellectual property (code) assets that were built.
Thank you for vociferousness! I recommend you check out EthStorage (video), which while you probably couldn’t implement this forum with it its web3:// , it doesn’t have the “pinning” problem and is a sincere effort.
Can you point out why its not a good concept for us?
You’re right, IPFS alone doesn’t guarantee availability, which is why pinning services are essential to keep data online. Many in the ecosystem combine pinning with multiple gateways or decentralized backup solutions. Tools like Sifflet can also help by monitoring data pipelines and ensuring your important data stays reliable and accessible.