It should have the same protections as an EULA, where the crawler is the end user, and crawlers should be required to read it and apply it.
It should have the same protections as an EULA, where the crawler is the end user, and crawlers should be required to read it and apply it.
So none at all? EULAs are mostly just meant to intimidate you so you won't exercise your inalienable rights.
I find that extremely hard to believe. Do you have a source?