They know how to actually block this stuff if they want to.

The issue is, it comes with costs, to things like SEO.

It's not hard to lock a computer system down so only logged in users can see content. It's a lot harder to lock them down so that only logged in users AND a select few entirely unauthenticated scraping tools can see your site to assist you in promoting it while OTHERWISE people aren't able to access it.