I suspect they simply so not care. Owners of these companies are exactly the sort of people that are genuinely puzzled and offended when someone wants them to think about anything but themselves

The attitude is visible in everything around AI, why would crawling be different?