I'm increasingly thinking that the only solution may be to have a search index with human editorial control. As much as I think Google deserves scorn for how badly they have made the search experience on the web, we have to remember that they're fighting a battle against SEO and spam. It feels like they have mostly given up. Any automatic solution will always be an arms race where they will be one step behind. The only way to get ahead is to ensure that all content has to pass through human moderation before it can be included -- that's the only way I can think of to break the cat and mouse game dynamics at play here.
Google, Facebook and others will never do this because it's expensive. Their cost savings is our toxic waste.
So what does a viable solution look like? I'm not sure. Maybe a system with distributed, volunteer moderators like Wikipedia. It's not perfect, but that model has held up remarkably well over its lifespan and has seemed to mostly hold its own against increasingly sophisticated attacks.
The scale of this would need to be unfathomable. Doesn’t YouTube get something like 100 years of video uploads every day? You’d need hundreds of thousands of people working full time to review that much footage manually.
Figuring out how to scale it may be a challenge, but it may be worth considering something along the lines of the directory sites that used to be the main method of discovery on the web prior to the rise of search engines, which itself can be searched.
I would personally find this useful at least, particularly when diving into unfamiliar topics. In a lot of those situations, a handful of high quality links can preempt a litany of more granular searches.
This might be an area where a traditional, respectful search engine style crawler backed by a small cheap LLM paired with human curation would work well. The crawler+LLM finds sites, attempts to compute some quality score, figures out where they'd fit in the directory's hierarchy, and then submits them for human review and approval. If a poor quality link makes its way through, users can flag it for review and if curation agrees it gets booted.
> Google deserves scorn for how badly they have made the search experience on the web, we have to remember that they're fighting a battle against SEO
Where did you get this idea from? They actively promote it.
>we have to remember that they're fighting a battle against SEO and spam
We have to remember that they decided to side with the SEOmongers rather than crush them, because it was better for Google's bottom line.
On one hand they fight SEO and spam, on the other they take advertising money to manipulate the order of search results...