That's what robots.txt is for.