Basic auth can keep the crawlers out, for one thing.