Wednesday, January 1, 2020

Introduction

This is a private blog, intended for personal use only.
I intend to use this blog as a tool to collect and record my thoughts on search engine technology as I intend to explore different possibilities of search engine operation, techniques and methodologies.

Although I am a Systems engineer I enjoy programming as a hobby to keep me mentally stimulated, I intend to build a basic search engine to learn not only better programming practices but also gain some insight into search engine operation.

It is not intended to at any stage go-live with the engine, the value of the project is the understanding and knowledge gained in the process.

Current search engine: Google

Thursday, July 2, 2009

WIP Update

The majority of the positive page analysis is complete, i am expanding the positive weighting schema at the moment while working in parallel on the scheduling component.

Once I have a handle on the weighting scheme, i will work more on the new algorithm, i will post more on this shortly but the idea i am exploring is using multiple search indices's based on different algorithms furthermore these being user selectable.

I have started to draft some of the negative weights, however i think i will leave these for later in the project. I feel once the scheduler is complete that a basic framework will be in place allowing me to easily update these components.

Wednesday, June 24, 2009

WIP Update

With the basic crawler process working well, I am focusing on the page analytics. This is a rather time consuming process that I expect I will be coming back to frequently.

I have some interesting ideas about a new approach to the search engine algorithms, ill post some more details as I progress. In the meantime, for a bit of light entertainment check out bingfail (found randomly on the internet) : BingFail

Tuesday, June 16, 2009

A case against the robot exclusion standard

While working on the crawler design, I have been thinking a bit about the robots exclusion standard. Initially I implemented a system to validate the request against a cached table of robot rules for each page then a thought hit me… why ?

Yes I understand the implications, netiquette, etc as well as the positive arguments & technical implications.

However assuming the project I am currently working on only crawls areas that a normal anonymous user would access I really see no point in implementing a robots method.

I do believe many of the larger search engines can be quite aggressive as to where & how they search through a site, not to mention Google for example has stated that their crawler can open over 100 requests concurrently to a single site during the crawl process.

Considering all the above & assuming a well behaved crawler wouldn’t this lead to a much better experience for the user ?

As my project is for education purposes with no commercial implications (heck it's not even going to go-live), I am going to investigate this approach with the goal being quite simple: What you see is what you get (WYSIWYG).

Sunday, June 14, 2009

WebCrawler prot-type up and running

A basic WebCrawler proto-type is now complete. At this stage the crawler simply fetches the web pages for later processing, this needs to be greatly enhanced to be of any use.

The crawler operates in a single thread at the moment, my plan is to expand it’s capacity as follows:
- Single threaded
- Multi threaded, single host
- Client-Server, Multi threaded, multi host

I will work on the scheduler mechanism as well as the processing shortly, I plan to have a queuing/scheduling mechanism separate from the processing & crawling. This should allow the system to correctly prioritise the work queue.

One point that was interesting, during the crawlers execution I noticed that many sites, including large corporate sites have very poor HTML coding standards (missing key HTML tags, html, head, body, etc). This will make the processing a bit more difficult in the next stage, I think I will add basic web page validity test as a negative weight when I am ready to work on the page processor.

Thursday, June 11, 2009

Page rank, a sterile corporate controlled internet ?

Touching back on a previous post, I highlighted that most new sites I find are via a recommendation from a friend or colleague. Thinking more about this, I was contemplating the last time I actually found a brand new site via a search engine.

The big jump in search engines came when Google went live, toppling the other search engines at the time with its much touted PagRank system returning what was purportedly more relevant search engine results with less spam.

The basic idea being that a trusted authority would be ranked highly, possibly affecting the rank of other sites it has links to. Bing itself uses a enhanced PageRank system.

However what is the cost of the PageRank system? How has it affected the internet and communities involved? It is certain that the PageRank system was a key driving factor in the then emerging SEO (search engine optimisation) industry.

Working regularly with the web optimisation team in my company, which deals with numerous SEO agencies to optimise our web applications I know full well what is entailed in time, effort & cost to ensure a website lists as high as possible in modern search engine rankings.

How is all this relevant?

Simply put, your rank in a PageRank based search engine depends on the amount of time, money & manpower you have available. Further to this, once achieved it is likely that within reason you will be able to maintain your ranking if you continue to invest in a proportional appropriate amount of resources.

Further to this, those sites with a high PageRank are likely to only link to sites within their own corporate domains.

The end result?

Perhaps the end result, is a search ecosystem returning a never ending list of expected results, stagnated by the same system that was intended to return relevancy.

Indexing the entire internet might be a virtuous goal, but to me at least it seems diametrically opposed to the current PageRank based search engines.

Sunday, June 7, 2009

Can't see the forest for the tree's

or aka Why can't I find anything?
Ever wonder why you can get thousands of results from a search engine, yet can't seem to find anything useful? There are a myriad of reason as to why this happens, for example over the past couple of years I have noticed in particular the google search engine results have been less accurate & relevant.

Traditionally this is considered the fate of all search engines, why? Are the goals of representing relative data to the user & indexing the greatest portion of the Internet mutually exclusive?

I have put together some basic tests, which anyone can reproduce highlighting some of the different approaches, strengths & weaknesses of some of the current engines.

Note: the results were accurate at the time of writing, due to the nature of the internet/search engine technology these results may differ.

search term A, ordered by total number of results
over-search, number of results for a non term

engine, results (lower is better)
www.google.com, 34360000000
www.altavista.com, 27800000000
www.yahoo.com, 26900000000
www.alltheweb.com, 17000000000
www.bing.com, 6410000000
www.lycos.com, 1344243025
www.ask.com, 1120000000
www.webwombat.com.au, 0

Is full-text indexing of the entire Internet really useful ? useful to who ? google or the end user ? Do we need 34,360,000,000 results for the letter A ?

search term “Bill Clinton”, ordered by number of subjective relevant results in first page (out of 10, but excluding Wikipedia)
subjective relevancy, number of results deemed useful

engine, results (higher is better)
www.alltheweb.com, 7
www.lycos.com, 6
www.altavista.com, 3
www.google.com, 3
www.bing.com, 2
www.ask.com, 2
www.yahoo.com, 2
www.webwombat.com.au, 0

This is a subjective test, in this case I was looking for useful information on the search term. Kudos to alltheweb for providing the most relevant & useful results & tsk tsk @ google for failing this one...

search term ladel, ordered by number of irrelevant results in first page (out of 10)
incorrect results, number of results that do not contain the search term at all.

engine, results (lower is better)
www.lycos.com, 7
www.google.com, 7
www.ask.com, 6
www.alltheweb.com, 6
www.yahoo.com, 5
www.altavista.com, 5
www.bing.com, 1
www.webwombat.com.au, 0

This clearly shows how some search engines possibly through the use of phonetics are actually out-smarting themselves. In this case lycos & google returned 7 out of 10 results which were not relevant (to the point of not even containing) the search query. Bing was quite impressive and the results returned were both accurate and relevant.

search term "Australia", ordered by rank of Wikipedia results in the first page (out of 10)
over-reliance on 3rd party information sources (inaccurate)

engine, uses wiki?, rnk of wiki, no. of wiki no.of s-relv
www.alltheweb.com, yes, 2, 3, 2
www.lycos.com, yes, 2, 3, 1
www.google.com, yes, 3, 1, 2
www.ask.com, yes, 4, 1, 3
www.yahoo.com, yes, 5, 1, 2
www.altavista.com, yes, 8, 1, 2
www.bing.com, no, n/a, 0, 7
www.webwombat.com.au, no, n/a, 0, 0

uses wiki?, is a Wikipedia result displayed in the first 10 results
rnk of wiki, how high (relevant) is the Wikipedia result
no. of wiki, number of references to Wikipedia or wiki's in general
no.of s-rel, number of subjective relative results (excluding Wikipedia)


Here I was interested as to how wiki has influenced search results, so here I am looking at how high wiki is ranked, how many wiki references then to see how many useful results were returned. Kudos again to bing which returned the most relevant results to the search term without resorting to what is (in this case) an atrocious and factually inaccurate wiki article (see how many errors you can find).

Overall the tests were interesting and showed some varied results which may give some insight as to why it is sometimes difficult to find information in a modern search engine.

Scope of project, AU & NZ only

After spending quite some time working on some basic design principles, I have decided to limit the scope of this project to AU & NZ websites only.

This will enable me to work on a controlled segment of the Internet, reducing the computational & bandwidth requirements while giving me a good sample size.

Interestingly enough, I have to decide just what an Australian (as an example) website is...
- a website physically located in Australia
- a website with a .au suffix
- a website that services Australia

for example, au.yahoo.com vs google.com.au.

I propose that an Australian website is either a site that ends in a .au suffix regardless of its geographical location. Further more that sites which use sub domains, eg. au.yahoo.com will not be considered.

In addition, any IP address based sites will considered based on their physical location via the use of a Geo-database.

Next up crawler basics, design & considerations.

Saturday, June 6, 2009

Why do people use search engines?

Reflecting about 'why' people use search engines, what are they actually searching for ?

People use search engines to find 'stuff'

How has 'Stuff' changed over time, in the early days of search engines, I remember using Infoseek, later Altavista (which i used for sometime) to find actual web sites.

These days, it seems with Google people are looking more & more for information?

In addition, people use search engines to find goods & services, I do this infrequently as once found I tend to stay with a particular service provider.

I guess quite allot these days, I will visit or possibly end up using a site via a recommendation by a colleague or friend.

Lastly, how does Internet sub-culture affect how and what we use search engines for ?

Is it possible to bring all these aspects together, coherently, in an accessible manner in a search engine... ?