If you use Cloudflare to proxy your site, there is a button to click that blocks the AI crawlers (even the free tier). It is almost as if the AI crawlers are a DDoS attack. You can't really do it any other way, since many don't respect robots.txt. At least until someone comes up with crowdsourced blacklists with few false positives.
"You can't really do it any other way"
Any custom solution by a half-competent programmer filters out all web crawlers. I'm running a semi-public website for years and nothing gets past