YaCy[0] is a P2P search engine. There's public instances but you can self-host it as well. It includes a spider to do your own crawling. It's challenging now between CloudFlare and other anti-bot CAPCHAS crawling effectively is really challenging. Bots (AI and otherwise) have ruined the web.