-- Crawler Halting
Hey Paul,
If it's not crawling anything within 5 to 10 minutes, it's probably not working. I have been doing some code clean up and fixes to some of the web scraper code that was committed since 3.3. I haven't been been doing much crawling recently, so hadn't realized some stuff was broken. Since my fixes I have started a crawl on an Ubuntu 16.04 LTS machine and it seems to be going well. If you have time, maybe get the most recent commit and try to see if it solves your problem.
Best,
chris
Hey Paul,
If it's not crawling anything within 5 to 10 minutes, it's probably not working. I have been doing some code clean up and fixes to some of the web scraper code that was committed since 3.3. I haven't been been doing much crawling recently, so hadn't realized some stuff was broken. Since my fixes I have started a crawl on an Ubuntu 16.04 LTS machine and it seems to be going well. If you have time, maybe get the most recent commit and try to see if it solves your problem.
Best,
chris