2012-05-03
Version 0.86 Released!.
Adds initial support for word suggestions as a user types in queries. The bigramming used
to speed common two word queries now works with n word grams. In addition, to being able
to build a bigram filter file using with complete Wikipedia dumps, one can also process the
smaller raw page count dumps which is faster. Adds support for * and $ in allow and
disallowed to crawl sites. Using this one can crawl sites to a fixed depth. Robots.txt processing
now supports the Google/Yahoo/Bing extension for * and $ in robot.txt paths. Support for
NOSNIPPET, NOARCHIVE, X-Robots-tag HTTP headers, has also been added. A tool for editing
search summaries after a crawl has also been added.
to speed common two word queries now works with n word grams. In addition, to being able
to build a bigram filter file using with complete Wikipedia dumps, one can also process the
smaller raw page count dumps which is faster. Adds support for * and $ in allow and
disallowed to crawl sites. Using this one can crawl sites to a fixed depth. Robots.txt processing
now supports the Google/Yahoo/Bing extension for * and $ in robot.txt paths. Support for
NOSNIPPET, NOARCHIVE, X-Robots-tag HTTP headers, has also been added. A tool for editing
search summaries after a crawl has also been added.
Adds initial support for word suggestions as a user types in queries. The bigramming used<br>to speed common two word queries now works with n word grams. In addition, to being able<br>to build a bigram filter file using with complete Wikipedia dumps, one can also process the <br>smaller raw page count dumps which is faster. Adds support for * and $ in allow and <br>disallowed to crawl sites. Using this one can crawl sites to a fixed depth. Robots.txt processing<br>now supports the Google/Yahoo/Bing extension for * and $ in robot.txt paths. Support for <br>NOSNIPPET, NOARCHIVE, X-Robots-tag HTTP headers, has also been added. A tool for editing<br>search summaries after a crawl has also been added.
(c) 2024 Yioop - PHP Search Engine