Has anyone tried making a p2p search engine? Naively it seems like one of the p2p networks from the late 90s or early 2000s should be able to handle it.
Searches propagate between nodes, nodes specialize on specific sections of the index, which can be random + user preference (maximizing redundance and availability for the things people actually care about).
Soulseek already gets like halfway there and it's not even trying to be. How hard could it be?
So my MS research focused on P2P networks, and with the caveat that the area has been out of favor (and hence under-researched) for about 2 decades, I would say it would be extremely hard to build even decent quality search in a purely decentralized manner.
There were essentially two main ways of doing P2P search: flooding and Distributed Hash Tables (DHT). Flooding is exactly how it sounds: you flood a query across the whole network. The most popular applications like Limewire, Kazaa etc were flooding based, and as you can imagine, didn’t scale well. I recall some stories of havoc on networks due to the amount of traffic they generated. However they were simple and allowed somewhat complex queries, because each node basically ran a brute force search on its local index.
On the other hand, DHTs organize nodes in a distributed index structure such that queries can be routed in a deterministic manner, typically in O(logN) hops. DHTs were basically the granddaddies of distributed NoSQL DBs like DynamoDB and Cassandra.
So you can already see the limitations of DHTs: they need keys to be indexed, typically keywords. You had to pick a small handful of keywords for it to be manageable. This inherently limits the complexity and scale of the queries.
Both had an existential problem with spam. There were essentially no defenses against bad actors poisoning the network with spam and malware. It was a huge unsolved problem until Bitcoin popularized proof of work (which had been suggested in various forms before, but Bitcoin was the first practical, robust application.)
And neither could approach the sophistication of Google, which could extract very high quality signals from their centralized troves of data to refine and rank results.
Personally though I think the time is ripe for a renaissance. Spam and bad actors are still existential problems, but with vectors / embeddings the discovery aspect could be addressed. It occurs to me that the most popular large scale vector search mechanism is HNSW networks, which seem pretty amenable to P2P networks…
There's yacy. I didn't get useful results wheb I tested it though.
https://yacy.net/
Back around 2008 it didn't give great results either. It's amazing how long the project manages to stay alive though
YaCy[0] is a P2P search engine. There's public instances but you can self-host it as well. It includes a spider to do your own crawling. It's challenging now between CloudFlare and other anti-bot CAPCHAS crawling effectively is really challenging. Bots (AI and otherwise) have ruined the web.
[0] https://yacy.net/index.html