Thinking about distributed (shared) caching?

[cottrell]

Just a question if anyone has given much thought on doing some sort of caching via IPFS, datproject or similar decentralized file-system. I am mostly curious about opinions on protocols/choices/trade-offs for communal scraping activities, not sure if the caching for this particular project would actually be useful given the sparsity of the data and number of typical requests.

[ipeirotis]

Having a shared cache seems to be a compelling feature, which is worth exploring.

Currently, we do not collect any usage data, so it is hard to estimate the effect of having such a cache.

[MoritzImendoerffer]

Great idea in my opinion. What about copyright infringements? Could it have financial consequences for the actors if we build up such a database?

[silvavn]

I think the main consequence would be dealing with G about infringing their terms. However, other tools, such as publish or perish, already allow you to do (private) caching, so might be worth exploring.

[cottrell]

My impression is there is always pushback against this kind of thing but there is very little ground. And the data really would be a cache since it could drift over time and you need to occasionally go back to the original.

[silvavn]

Well, in an extreme case, instead of collecting data from G, we could collect them directly from the sources (ieee, acm, etc)

[Charlotte-br560]

Considering distributed caching? I've pondered leveraging IPFS, datproject, or similar decentralized file systems for caching. It's an intriguing idea, especially for communal scraping endeavors. The choice of protocol boils down to trade-offs - for instance, IPFS offers robustness but requires managing content addressing, while datproject emphasizes versioning. As for the usefulness in your project, it depends on data density and request frequency. Sparse data might not benefit greatly, but it's worth exploring for efficiency and scalability.

[cottrell]

FWIW I managed to get some caching experiment running with IPFS or BTFS and then was fiddling with some libp2p-pubsub based sharing mechanism.

It's been a while but I think the sharing of pubkeys was the main thing on the libp2p side.