08/01/2022

User Icon

Notices: in yioop crawls, index, menu, XML crawls, issue .

Hey Chris, I tried to understand the project getting errors, warnings, can you help here:
1) Hamburger menu is not overlaying on top, it's sliding the webpage on rightside instead popupmenu any solution?
2)Media updater and Queueserver, Fetcher automatically gets to yellow indicator after 15 minutes while crawling?
3)In manage crawls, When one row is "set as index" other rows are not indexed, seems only one index can be active at a time not multiple?
4)'Image' category not showing any images after crawl of sample pics website, just showing web results in it any fixes?
5)Some crawl just following robots.txt directives has only sitemapindex.xml which has several sitemaps. How do we crawl such website?
(Edited: 16/01/2022)
Hey Chris, I tried to understand the project getting errors, warnings, can you help here: 1) Hamburger menu is not overlaying on top, it's sliding the webpage on rightside instead popupmenu any solution? 2)Media updater and Queueserver, Fetcher automatically gets to yellow indicator after 15 minutes while crawling? 3)In manage crawls, When one row is "set as index" other rows are not indexed, seems only one index can be active at a time not multiple? 4)'Image' category not showing any images after crawl of sample pics website, just showing web results in it any fixes? 5)Some crawl just following robots.txt directives has only sitemapindex.xml which has several sitemaps. How do we crawl such website?
24/01/2022

-- Notices: in yioop crawls, index, menu, XML crawls, issue
Hi Kas,
On
(1) Maybe add a not too large screenshot.
(2) This tells me the processes probably didn't get started correctly which might be caused by some pre-req (such as being able to due popen not being satisfied) Was these anything in the error_logs?
(3) "Set as index" is used to specify the default index to serve search results from, it does not have anything to do with what crawls are active.
(4) Without some details on what you were doing it is hard for me to debug.
(5) Not sure what you are asking. If during a crawl yioop discovers a robots.txt file with a link to a sitemap, it will try to download the sitemap.
Best,
Chris
Hi Kas, On (1) Maybe add a not too large screenshot. (2) This tells me the processes probably didn't get started correctly which might be caused by some pre-req (such as being able to due popen not being satisfied) Was these anything in the error_logs? (3) "Set as index" is used to specify the default index to serve search results from, it does not have anything to do with what crawls are active. (4) Without some details on what you were doing it is hard for me to debug. (5) Not sure what you are asking. If during a crawl yioop discovers a robots.txt file with a link to a sitemap, it will try to download the sitemap. Best, Chris
26/01/2022

User Icon
-- Notices: in yioop crawls, index, menu, XML crawls, issue
Thanks chris just on semester, my classmate uses sphider for his project and it's crawling images from any website and it shows in images tab bunch of pictures. I am trying with yioop for images to work like Yandex but it shows like as a hyperlink textformat.png filename of image not picture what setting to use?
To the previous one: (3)Q#. crawl01 made set as index, crawl02, crawl03 are not, only one crawl is getting active.
(5)Q#. It's following only robots.txt and showing sitemap.html as text links in search results not the webpages of a site
(Edited: 26/01/2022)
Thanks chris just on semester, my classmate uses sphider for his project and it's crawling images from any website and it shows in images tab bunch of pictures. I am trying with yioop for images to work like Yandex but it shows like as a hyperlink textformat.png filename of image not picture what setting to use? To the previous one: (3)Q#. crawl01 made set as index, crawl02, crawl03 are not, only one crawl is getting active. (5)Q#. It's following only robots.txt and showing sitemap.html as text links in search results not the webpages of a site
06/02/2022

-- Notices: in yioop crawls, index, menu, XML crawls, issue
How about a small screenshot of your configure activity to convince me that it is not giving a warning about failed pre-reqs? Then maybe let me know what version of Yioop you installed, what version of php you have etc.
Best,
Chris
How about a small screenshot of your configure activity to convince me that it is not giving a warning about failed pre-reqs? Then maybe let me know what version of Yioop you installed, what version of php you have etc. Best, Chris
14/03/2022

User Icon
-- Notices: in yioop crawls, index, menu, XML crawls, issue
Hi chris, On (1) Maybe add a not too large screenshot. (attachmend scrn menu disturbs background) Resource Description for Screenshot_20220322-202238~2.png
(2)Where to check the popen() for the same and manage machine cant start the server and fetcher via GUI? using terminal to start and stop.
(3)Tested some news site it has full of sitemaps as checked in robots.txt, how to crawl such sites which has sitemap_index.xml under it again several sitemaps.
(4) how do we crawl images site lets say pixabay, here is the config page details as you asked lastime and log file nothing its showing empty.
Search Engine Work Directory localhost/serp_project/work_directory [Load/Create] Component Check Check Passed. Profile Settings Default Language:English Debug Display #Error Info #Query Info #Test Info
(Edited: 22/03/2022)
Hi chris, On (1) Maybe add a not too large screenshot. (attachmend scrn menu disturbs background) ((resource:Screenshot_20220322-202238~2.png|Resource Description for Screenshot_20220322-202238~2.png)) (2)Where to check the popen() for the same and manage machine cant start the server and fetcher via GUI? using terminal to start and stop. (3)Tested some news site it has full of sitemaps as checked in robots.txt, how to crawl such sites which has sitemap_index.xml under it again several sitemaps. (4) how do we crawl images site lets say pixabay, here is the config page details as you asked lastime and log file nothing its showing empty. Search Engine Work Directory localhost/serp_project/work_directory [Load/Create] Component Check Check Passed. Profile Settings Default Language:English Debug Display #Error Info #Query Info #Test Info
27/03/2022

-- Notices: in yioop crawls, index, menu, XML crawls, issue
Just giving a heads up that saw the updated message. I will try to recreate (1) and fix.
I am not sure I understand the question (2). However, if you are looking for where the popen is called to start a process used in crawling, it is in the file src/library/CrawlDaemon, in the method execInOwnProcess.
(3) Was this in version 8.0.0 or using the master git branch? I've noticed some crawl bugs in master branch that am fixing, probably within a day or two.
(4) I will try to test pixabay and see what's up.
Your config details look fine. I will post again to your thread by Mar 30 at the latest with some follow up.
Best,
Chris
Just giving a heads up that saw the updated message. I will try to recreate (1) and fix. I am not sure I understand the question (2). However, if you are looking for where the popen is called to start a process used in crawling, it is in the file src/library/CrawlDaemon, in the method execInOwnProcess. (3) Was this in version 8.0.0 or using the master git branch? I've noticed some crawl bugs in master branch that am fixing, probably within a day or two. (4) I will try to test pixabay and see what's up. Your config details look fine. I will post again to your thread by Mar 30 at the latest with some follow up. Best, Chris
01/04/2022

User Icon
-- Notices: in yioop crawls, index, menu, XML crawls, issue
Resource Description for Screenshot_20220331-005633~2.png Chris, yeah I mean the thread errors refering to v7.4 and cent os tried to understand issues while project. The menu is not opening on-top of searchpage its sliding the wholepage itself compressing webpage. Eg. google, yandex menu opens on-top without disturbing background. I think yioop just needs good Hardware n memory to crawl largest indexes otherwise its a very good project next to Java,C/C++
 
(Edited: 01/04/2022)
((resource:Screenshot_20220331-005633~2.png|Resource Description for Screenshot_20220331-005633~2.png)) Chris, yeah I mean the thread errors refering to v7.4 and cent os tried to understand issues while project. The menu is not opening on-top of searchpage its sliding the wholepage itself compressing webpage. Eg. google, yandex menu opens on-top without disturbing background. I think yioop just needs good Hardware n memory to crawl largest indexes otherwise its a very good project next to Java,C/C++

-- Notices: in yioop crawls, index, menu, XML crawls, issue
Hi Kas,
For (1) I see what you are saying and will try to come up with a CSS fix. For (3) above I have fixed some issues related to crawling in the current master branch, although I've found a couple of new ones that I will work on (I currently debugging a Greek stemmer, which may take a couple days). The image search investigation is still on my to-do list, but I haven't had a chance to look at.
Hi Kas, For (1) I see what you are saying and will try to come up with a CSS fix. For (3) above I have fixed some issues related to crawling in the current master branch, although I've found a couple of new ones that I will work on (I currently debugging a Greek stemmer, which may take a couple days). The image search investigation is still on my to-do list, but I haven't had a chance to look at.
04/04/2022

User Icon
-- Notices: in yioop crawls, index, menu, XML crawls, issue
Sure Chris, as a project I was looking at yandex, and trying to understand what is tht countries dropdown actually is it in yioop too. Then observed most urls looking like this sitexyz.com/?q= or sitexyz.com/search?q= but yioop shows like /its=12345678/?q= can we do it masking url to yioop.com i'm learning but its interesting topic for books crawl but thumbnail issue I think images, books have same functions of previewing and url of image or book
(Edited: 04/04/2022)
Sure Chris, as a project I was looking at yandex, and trying to understand what is tht countries dropdown actually is it in yioop too. Then observed most urls looking like this sitexyz.com/?q= or sitexyz.com/search?q= but yioop shows like /its=12345678/?q= can we do it masking url to yioop.com i'm learning but its interesting topic for books crawl but thumbnail issue I think images, books have same functions of previewing and url of image or book
09/04/2022

-- Notices: in yioop crawls, index, menu, XML crawls, issue
In the git repository, I have now fixed the hamburger menu functionality so it doesn't scrunch up the text. I used yandex a while back to translate some of the static strings on this site to whichever language I was translating to. The yandex tool for this said I needed to put a an acknowledgment link for the translation, so I did. The site doesn't use yandex for search. Queries with its=some number, tell Yioop what index to serve the search results from. If you don't have it, Yioop will serve search results from whatever is its current default index.
Best,
Chris
In the git repository, I have now fixed the hamburger menu functionality so it doesn't scrunch up the text. I used yandex a while back to translate some of the static strings on this site to whichever language I was translating to. The yandex tool for this said I needed to put a an acknowledgment link for the translation, so I did. The site doesn't use yandex for search. Queries with its=some number, tell Yioop what index to serve the search results from. If you don't have it, Yioop will serve search results from whatever is its current default index. Best, Chris
X