After spending quite some time working on some basic design principles, I have decided to limit the scope of this project to AU & NZ websites only.
This will enable me to work on a controlled segment of the Internet, reducing the computational & bandwidth requirements while giving me a good sample size.
Interestingly enough, I have to decide just what an Australian (as an example) website is...
- a website physically located in Australia
- a website with a .au suffix
- a website that services Australia
for example, au.yahoo.com vs google.com.au.
I propose that an Australian website is either a site that ends in a .au suffix regardless of its geographical location. Further more that sites which use sub domains, eg. au.yahoo.com will not be considered.
In addition, any IP address based sites will considered based on their physical location via the use of a Geo-database.
Next up crawler basics, design & considerations.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment