As a site owner, how does one opt out of this, since it obviously ignores robots.txt?
Shoot me your site and I can blacklist it
That's what robots.txt is for.
Shoot me your site and I can blacklist it
That's what robots.txt is for.