While working on the crawler design, I have been thinking a bit about the robots exclusion standard. Initially I implemented a system to validate the request against a cached table of robot rules for each page then a thought hit me… why ?
Yes I understand the implications, netiquette, etc as well as the positive arguments & technical implications.
However assuming the project I am currently working on only crawls areas that a normal anonymous user would access I really see no point in implementing a robots method.
I do believe many of the larger search engines can be quite aggressive as to where & how they search through a site, not to mention Google for example has stated that their crawler can open over 100 requests concurrently to a single site during the crawl process.
Considering all the above & assuming a well behaved crawler wouldn’t this lead to a much better experience for the user ?
As my project is for education purposes with no commercial implications (heck it's not even going to go-live), I am going to investigate this approach with the goal being quite simple: What you see is what you get (WYSIWYG).
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment