Tuesday, June 16, 2009

A case against the robot exclusion standard

While working on the crawler design, I have been thinking a bit about the robots exclusion standard. Initially I implemented a system to validate the request against a cached table of robot rules for each page then a thought hit me… why ?

Yes I understand the implications, netiquette, etc as well as the positive arguments & technical implications.

However assuming the project I am currently working on only crawls areas that a normal anonymous user would access I really see no point in implementing a robots method.

I do believe many of the larger search engines can be quite aggressive as to where & how they search through a site, not to mention Google for example has stated that their crawler can open over 100 requests concurrently to a single site during the crawl process.

Considering all the above & assuming a well behaved crawler wouldn’t this lead to a much better experience for the user ?

As my project is for education purposes with no commercial implications (heck it's not even going to go-live), I am going to investigate this approach with the goal being quite simple: What you see is what you get (WYSIWYG).

Sunday, June 14, 2009

WebCrawler prot-type up and running

A basic WebCrawler proto-type is now complete. At this stage the crawler simply fetches the web pages for later processing, this needs to be greatly enhanced to be of any use.

The crawler operates in a single thread at the moment, my plan is to expand it’s capacity as follows:
- Single threaded
- Multi threaded, single host
- Client-Server, Multi threaded, multi host

I will work on the scheduler mechanism as well as the processing shortly, I plan to have a queuing/scheduling mechanism separate from the processing & crawling. This should allow the system to correctly prioritise the work queue.

One point that was interesting, during the crawlers execution I noticed that many sites, including large corporate sites have very poor HTML coding standards (missing key HTML tags, html, head, body, etc). This will make the processing a bit more difficult in the next stage, I think I will add basic web page validity test as a negative weight when I am ready to work on the page processor.