Regardless. You should still respect robots.txt..