> Common Crawl, with over one billion, nine hundred and seventy thousand web pages in their archive: 345TB.
Common Crawl is 300 billion webpages and 10 petabytes. I suppose your number is 1 of our 122 crawls.
> Common Crawl, with over one billion, nine hundred and seventy thousand web pages in their archive: 345TB.
Common Crawl is 300 billion webpages and 10 petabytes. I suppose your number is 1 of our 122 crawls.
oh, i didn't see that the 1.97 billion pages were crawled in a 11 day period earlier this month. either way, nearly 2,000,000,000 pages fit in ~third of a petabyte...
p.s. thanks for correcting me, i was using this information for something else, and now it's correct!