Search Engines: Where We Were, Are Now, and Will Ever Be
Unfortunately, I was unable to contribute to the decennial issue at the editors' invitation due to a family bereavement, but since it was such a good idea to take a look back at where we were, and then relate it to the present day and beyond, I did not want to miss the opportunity in this issue.
Where Were We?
When starting any kind of retrospective, the first place to visit is always going to be the Wayback Machine, or the Internet Archive [1] as it is also known. The immediate temptation is to see what Google was like ten years ago, and it comes as something of a shock to realise that it did not make its debut for another couple of years. The search engine of choice for many people then was AltaVista [2], and looking at early iterations of the home page are actually quite shocking in many ways - while it fits onto one screen the graphics are very poor, a large advert dominates the page and the search options are limited to searching the web or USENET and three display formats are offered. An advanced search option was offered, but this was also fairly limited. The other two 'options' on the page (if indeed they can be called that) are 'Contests' and 'Create a site'.
It's a very untidy looking page, and if anyone produced such a page today the kindest description that could be given would be 'amateurish'.
What, however, is interesting about the page is not what it says, but what we can divine from it - of course, hindsight is a wonderful thing, but nonetheless the impression is that AltaVista was just an experiment (which is not far from the truth), and not to be taken particularly seriously, yet at the time we were amazed at what it was able to do for us. Ironically however, some of the options, such as limiting a search to upper or lower case have since disappeared, and it is doubtful that they will return. In those far-off days the simple fact that a search engine could index Web pages and return them to us in a list was quite remarkable, and we gave little thought to Web page design, and were happy to accept whatever we could get, however poor (with twenty-first century vision) it was.
Yahoo! [3], which was the other major contender for the king of search engines, was also very basic, with poor graphics and few options. The main emphasis on the page was of course the directory or index element, with a dozen options available, which interestingly enough have not really changed to the present day although of course the emphasis placed upon them has, quite dramatically. The overall sense that I got from both sites (which is of course just my impression) is that both companies were taking the approach of 'we don't know what we're doing, but it looks promising, so we'll see where we go'.
Over the next few months, from mid-1997, we began to see the advent of the search engine wars, which were largely fought out over the size of their indexes, and there's an interesting article on that by Danny Sullivan [4]. More search engines began to appear, notably Northern Light, which was a pioneer in the field of clustering and query refinement. The buzzword in the mid-1990s was 'portal', as the search engines began to realise that they needed to do rather more than simply offer a search service, and Yahoo!, Lycos, Excite and Infoseek began experimenting with the concept of adding more services and greater value for their users. At that particular time it was quite hard to get a search engine to admit that that was what it was - they were all emphasising the fact that search was only a part of what they were doing. However, this in turn changed with the dot.com bust of 2000, with many engines disappearing, while the rest tried to recoup as best they could and return to the basic core structure of their business.
Google itself did not appear until late 1998 in any significant form [5], and the early home pages really underline the concept of simple, clear and uncluttered search engine interfaces. During the course of 1999 Google experimented with slightly different designs, but the home page has remained remarkably consistent over the 8 years of its development. More importantly however, the importance of ranking, and how to achieve a good ranking with the search engines moved to centre stage, and increased the need to get on that elusive first page. At the end of 1999, Google introduced advertising linked to searches, and set other search engines on the same path. But I still find it strange to think that something so obvious and now so ubiquitous took such a long time to develop; today we think nothing of this, but towards the turn of the century it was still something quite novel.
In many respects, while search engines continued to expand, finding new revenue streams and creating a niche for themselves, very little work was actually done on the search and retrieval options that users were offered. It was as true then as it is today that most users are reasonably satisfied with simple quick and dirty searches, rather than more in-depth and detailed strategies. Rather than improving existing facilities, some search engines began to take a slightly different route with natural language search and direct answers to specific questions, which again was quite revolutionary, but which is now an aspect of search that is incorporated into the major search engines and something users expect to find as a given.
Since 2000 Google has continued to increase its dominance in the search engine market place, but ironically I feel that it's managed to do this at least in part because it has ignored search itself and diversified into many other areas. Indeed, if you look at its mission statement 'to organize the world's information and make it universally accessible and useful.' [6], it does not actually mention search at all. Rather, Google is in the business of audience aggregation, and with a few notable exceptions it develops into areas where it can attract and also maintain an audience. Blogger [7], Gmail [8], a personalised home page and so on are increasingly integrated. Other major search engines are also doing exactly the same thing themselves with both Yahoo! and MSN producing their own functions and facilities designed to catch, keep and grow an audience, with a clear aim of making money from them through advertising.
While the big three, or four, if you include Ask [9], are concentrating on diversifying and building up their markets, this is not leading to a monopoly - at least in terms of numbers. Hardly a day goes by that I do not discover a new search engine - in April alone I have written about 8 new search engines or significant developments to smaller existing engines in my weblog [10], and there are a number of smaller engines (at least in terms of audience) which are continually innovating with new functionality, such as Exalead [11] and Accoona [12]. However, one particular victim of search engine evolution seems to be the index or directory type engine. Yahoo! has relegated its directory on its home page to the extent that it is almost invisible now, and DMOZ, or the Open Directory Project [13] is seldom talked about. The only area in which this type of engine is still thriving is in country or regional search engines, and there are still a good two thousand or more of these available. Meta or multi search engines (engines that draw results from a variety of other search engines, de-duplicate the results and then display them) are continuing to thrive.
'Traditional' search engines are also facing a slight threat from some of the Web 2.0- based applications. There is a strong trend now towards the personalisation of results, which actually started some years ago, as users realised that the results they were obtaining from engines were not always the most appropriate ones for them. The big three in particular have worked hard in this area, Google with its personalised home page, MSN with the search builder refine results option, and Yahoo! with the Mindset [14] approach. However, applications such as Rollyo [15], Eurekster swicki [16] and Squidoo [17] are all offering users the ability to create their own search tools and resources to a much greater extent than ever before. While I do not think this is going to be a major concern for search engines in the near future, it does help to point to possible new developments with which they will have to contend.
Emerging Trends
Search engines are moving into the 'local' area, by offering tailored results and resources based on the physical location in which an enquirer happens to be. If we add mobile communication into the mix, the entire concept of search engines begins to take an interesting new twist. While making prophecies about the future is always risky, and doubly so when talking about the Internet it does not take a genius to work out that being in constant or semi-constant contact with a search engine is going to have considerable appeal to many people. If you are in a different city, and need a hotel for the night, it is not going to be too difficult for the search engines to work out where you are, based on GPS and your mobile phone and recommend an appropriate hotel. If your search is personalised, the search engine will be able to learn that you prefer hotels within a particular budget range and to return those results to you first. Adding in the mapping features that the big search engines already have, they can also guide you directly to the front door. News services will constantly be updated and the search engine could quite easily then inform you that your favourite band was playing in the same city the next evening, and you would be able to book a ticket there and then (with the search engine taking a suitable fee of course). If we then complicate things even further, the increased use of RFID (Radio Frequency Identification) chips will mean that a visit to a supermarket will become interesting, as the mobile device will be able to recognise products, interrogate a search engine database and tell you if the item you want to buy is cheaper down the road. As search engines are now getting involved in the entertainment industry with features such as video search, it no longer represents a great stretch of the imagination to assume that they will also want to tie in with supermarket chains. This then opens up huge commercial possibilities, since you will be able to tell your personalised search engine what you want to eat for supper, and by talking to the mobile device via the supermarket you can be directed around the store to pick up each of the ingredients - that is, if you prefer to do it manually, rather than sitting at home and ordering online through your Google Supermarket interface.
Conclusion
Consequently, search engines will become more pervasive than they already are, but paradoxically less visible, if you allow them to personalise increasingly your own search experience. Alerting services will become even more commonplace as search engines learn your interests and preferences and can inform you of new developments, news, new Web sites and so on without users having to do anything at all, since the engines can monitor what you do and where you go. Of course, this could easily turn into some form of Orwellian dystopia, so I think that users will need to consider very carefully exactly to what extent they let search engines into their lives. What is certain however is that the future will grow increasingly interesting and search engines in one form or another will be with us for a very long time to come.
References
- The Internet Archive http://www.archive.org/
- Altavista http://www.altavista.com
- Yahoo http://web.archive.org/web/19961017235908/http://www2.yahoo.com/
- Danny Sullivan, The Alta Vista Size Controversy, July 1997, SearchEngineWatch
http://searchenginewatch.com/sereport/article.php/2165321 - Google circa 1998 http://web.archive.org/web/19981202230410/http://www.google.com/
- Google's mission statement http://www.google.com/corporate/
- Blogger http://www.blogger.com
- Gmail http://www.gmail.com
- Ask http://www.ask.com
- Phil Bradley's weblog http://www.philbradley.typepad.com/
- Exalead http://www.exalead.com
- Accoona http://www.accoona.com/
- DMOZ http://www.dmoz.org/
- Yahoo! Mindset http://mindset.research.yahoo.com/
- Rollyo http://www.rollyo.com
- Eurekster swicki http://swicki.eurekster.com/
- Squidoo http://www.squidoo.com