2015-10-16

1 Billion Pages.

On Oct 15, 2015, Yioop completed its first billion page crawl. It took 1 year and 5 days to carry out. Of that time though Yioop was stopped for about 2 - 3 month with various upgrades and back-ups. Crawling speed varied during the crawl -- some of the changes made to Yioop slowed it down, some speeded it up. At its slowest it was going about 120-150 thousand pages an hour, at its fastest it was going about 300 thousand pages an hour. In the last week of crawling it was averaging 200 thousand pages downloaded per hour. Yioop consists of six, 2011 Mac Mini each with an 8TB external hard drive attached. The operating system was upgraded twice during the crawl and is currently running El Capitan.
On Oct 15, 2015, Yioop completed its first billion page crawl. It took 1 year and 5 days to carry out. Of that time though Yioop was stopped for about 2 - 3 month with various upgrades and back-ups. Crawling speed varied during the crawl -- some of the changes made to Yioop slowed it down, some speeded it up. At its slowest it was going about 120-150 thousand pages an hour, at its fastest it was going about 300 thousand pages an hour. In the last week of crawling it was averaging 200 thousand pages downloaded per hour. Yioop consists of six, 2011 Mac Mini each with an 8TB external hard drive attached. The operating system was upgraded twice during the crawl and is currently running El Capitan.
X