Robots.txt are also ToS of sorts.