I checked all in page options (including unknown), trying it again now.
In page Options, I ran the test again using URL for https://search.example.com/manual and got the redirections, then i tried the same on https://search.example.com/manual/index.html and got the warnings again, checked Apache logs and it says permission denied when the manuals folder has RW permissions for everybody, this is not making any sense to me.
Just as a sanity check can you view the web page you are trying to crawl in a browser using the developer mode and print the HTTP messages you see? Or from the command line use curl -v url_you_are_trying_to_crawl ? You keep mentioning read write permissions rather than saying that you saw 403 forbidden messages, so I just want to actually see what's going on.
Chris
I browse those sites all the time and I get the pages in them, but they are a lot of pages so I wanted to put a search engine on them so I can find easily what I'm looking for, I thought Yioop would help, but it hasn't so I just deleted it from the web server, you can count this as closed.