OTOH: ByteDance intern responsible for spamming your web server with crawlers that ignore robots.txt given permanent position with a raise, now in management.
OTOH: ByteDance intern responsible for spamming your web server with crawlers that ignore robots.txt given permanent position with a raise, now in management.
honoring robots.txt is an informal courtesy, not international law.
Not breaking the law is just about the lowest bar you can set for an organization.
We can go lower
FYI, we're still not sure whether the scraped AI training datasets involve copyright infringement.