Tuesday, June 16, 2009

A case against the robot exclusion standard

While working on the crawler design, I have been thinking a bit about the robots exclusion standard. Initially I implemented a system to validate the request against a cached table of robot rules for each page then a thought hit me… why ?

Yes I understand the implications, netiquette, etc as well as the positive arguments & technical implications.

However assuming the project I am currently working on only crawls areas that a normal anonymous user would access I really see no point in implementing a robots method.

I do believe many of the larger search engines can be quite aggressive as to where & how they search through a site, not to mention Google for example has stated that their crawler can open over 100 requests concurrently to a single site during the crawl process.

Considering all the above & assuming a well behaved crawler wouldn’t this lead to a much better experience for the user ?

As my project is for education purposes with no commercial implications (heck it's not even going to go-live), I am going to investigate this approach with the goal being quite simple: What you see is what you get (WYSIWYG).

No comments:

Post a Comment