It would be interesting to see a decentralised, residential collective that builds and publishes an index. There are surely enough interested people on HN alone that would be willing to run software at home to scrape a small slice of the internet.

Something like YaCy?

Just like that! I will give it a try

residential proxies are one of the leading causes of what's killing the internet right now