A basic WebCrawler proto-type is now complete. At this stage the crawler simply fetches the web pages for later processing, this needs to be greatly enhanced to be of any use.
The crawler operates in a single thread at the moment, my plan is to expand it’s capacity as follows:
- Single threaded
- Multi threaded, single host
- Client-Server, Multi threaded, multi host
I will work on the scheduler mechanism as well as the processing shortly, I plan to have a queuing/scheduling mechanism separate from the processing & crawling. This should allow the system to correctly prioritise the work queue.
One point that was interesting, during the crawlers execution I noticed that many sites, including large corporate sites have very poor HTML coding standards (missing key HTML tags, html, head, body, etc). This will make the processing a bit more difficult in the next stage, I think I will add basic web page validity test as a negative weight when I am ready to work on the page processor.
Sunday, June 14, 2009
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment