Because crawlers aren't concerned about the bandwidth of the sites they crawl and will simply continue to take everything, everywhere, all the time regardless of what sites do.

Also it's unfair to expect every small site to put in the time and effort to, in essence, pay the Danegeld to AI companies just for the privilege of their continued existence. It shouldn't be the case that the web only exists to feed AI, or that everyone must design their sites around feeding AI.