2014-09-01

Starting a New Larger Scale Crawl.

I finally have hardware in place and have concluded preliminary testing on the current version of Yioop so that I can do a new larger scale which I am starting today. My goal is something larger than my current record of 1/3billion pages crawled and indexed. Yioop still consists of six Mac Minis, but now each Mini has 8TB attached to it -- twice what it had before. Since the last major crawl a new summarizer has been added to Yioop, there has been improved support for Office documents and epub, and several new stemmers have also been added. Hopefully, this will be a cool crawl!
I finally have hardware in place and have concluded preliminary testing on the current version of Yioop so that I can do a new larger scale which I am starting today. My goal is something larger than my current record of 1/3billion pages crawled and indexed. Yioop still consists of six Mac Minis, but now each Mini has 8TB attached to it -- twice what it had before. Since the last major crawl a new summarizer has been added to Yioop, there has been improved support for Office documents and epub, and several new stemmers have also been added. Hopefully, this will be a cool crawl!
X