Must a history of search engines be dominated by Google?
What would the Yorkshireman who built the first truly modern search engine say? And what say searchers in China, South Korea, Russia, or the Czech Republic? Won’t they have an opinion?
And how about if we ask Jeeves?
What is the opinion of the new female CEO of one of Asia’s biggest search engines?
Human, algorithm, bot: welcome — it seems we’re all searching for something.
Who Invented the Search Engine?
Arguably the first search engine, Archie (or Archive without the ‘v’ and not a comic book character) was built by Alan Emtage, Bill Heelan and J. Peter Deutsch while they were computer science students at McGill University in Montreal, Canada.
It started as a local network-based searchable database of file names of File Transfer Protocol (FTP) archives.
The first commercial version of Archie—a complete rewrite of the software—was made available in October of 1992 by Emtage and Deutsch who had by then founded Bunyip Information Systems.
Used for searching Gopher protocol servers, Veronica was built by Steven Foster and Fred Barrie while at the University of Nevada in the US. Gopher is a read-only network file system consisting of hierarchical hyperlinkable menu items (similar to a web page) and was invented by Mark McCahill at the University of Minnesota in the US.
Veronica provided keyword search of a database of menu items from files on Gopher servers. Named after a character, Veronica Lodge, created by US comic book publisher Archie Comics.
Developed by Rhett Jones at the University of Utah in the US, Jughead also used the Gopher protocol but would search only a single server at a time.
And yes, it was also named after a character created by Archie Comics.
Built by the inventor of the world wide web and Londoner Tim Berners-Lee.
It was the first catalogue or directory of the world wide web and comprised of topic experts maintaining lists of links to quality resources.
You can still browse a version of The WWW Virtual Library today.
World Wide Web Wanderer
Developed by Matthew Gray, it was the first bot ever built and was designed to measure the growth of the web by counting active web servers.
It created one of the first indexes called the Wandex and was eventually upgraded to include URLs.
Created by Oscar Nierstrasz at the University of Geneva, it mirrored web resources linked to by high-quality web directories, reformatted the content and provided front-end dynamic querying.
Dutch software engineer Martijn Koster’s early search engine did not use a bot instead relying on website owners to submit pages they wanted indexing.
Koster also created the Robots Exclusion Standard which allowed website owners to block access to bots at a domain or individual page level.
The first complete search engine to use bots to build and maintain an index made up of document titles and headings and offer a web form for users to search the index using keywords — how modern search engines work.
Created by Jonathan Fletcher from Scarborough, North Yorkshire while working at the University of Stirling in Scotland.
Eventually abandoned due to lack of investment and server space(!).
World Wide Web Worm
Developed by Oliver McBryan at the University of Colorado in the US, it was the first implementation of a search engine which used the anchor text in a backlink to characterise the page it points to.
It would eventually be cited as an influence in Larry Page’s patent for PageRank.
Jerry and David's Guide to the World Wide Web or Yahoo!
US-based Stanford University students David Filo and Jerry Yang start their own directory formed from their favourite web pages.
Sensibly re-named Yahoo! —after that race of brutish creatures from Swift’s Gulliver’s Travels— it quickly expanded and uniquely offered a human authored description with each url.
Today Yahoo! is still a thriving brand, powered by Microsoft’s Bing and owned by US multinational telecoms giant Verizon.
Built by Brian Pinkerton WebCrawler was the first search engine to use bots to crawl and index entire web pages which has become the standard ever since. Users could search for any word on any webpage on the world wide web.
Bought by America Online in 1995 who would then sell WebCrawler to Excite in 1997. Eventually acquired by InfoSpace in 2001 when Excite (@Home) went bankrupt.
Today WebCrawler is a meta search engine blending results from Google, Yahoo! and ‘other popular search engines’.
After starting as a research project by Michael Mauldin Lycos launched with a catalogue of 54,000 documents.
It provided results based on ranked relevance retrieval, prefix matching and word proximity.
Lycos became the fastest IPO from inception to offering in 1996. Bought by South Korea-based Daum Communications in 2004 who then sold Lycos to India-based Ybrant Digital in 2010.
Founded by Steve Kirsch, Infoseek started as a pay-for-use service but quickly became free post-launch. It uniquely allowed webmasters to submit a page for listing in real-time.
It became the default search provider for the then popular Netscape browser in 1995 and following this went public 1996.
Infoseek was bought by Walt Disney in 1998 and became Go.com. The technology behind Infoseek was then eventually bought by Inktomi in 2000.
Ex-Infoseek engineer Robin Li co-founded (and is current Chairman and CEO of) the China-based search engine Baidu.
Architext or Excite
Using statistical analysis of word relationships, Architext was a project by Graham Spencer, Joe Kraus, Mark VanHaren, Ryan McIntyre, Ben Lutch and Martin Reinfried all from Stanford University in the US.
Excite was formally launched in 1995 with an IPO quickly following in 1996. Bought by broadband provider @Home in 1999. Bankrupt in 2001 and acquired by InfoSpace.
Based on an initial idea by Paul Flaherty; Louis Monier wrote the web crawler and Michael Burrows the indexer.
It was the first search engine to allow natural language queries, had unlimited bandwidth and allowed users to add or delete a url within 24 hours.
Altavista was bought by Overture in 2003 which was then bought by Yahoo! in the same year.
Homebase or LookSmart
Founded in Melbourne, Australia, Homebase was majority owned by Reader’s Digest but was bought back from them by original founders Evan Thornley and Tracey Ellery; re-located to California and re-branded as LookSmart; signed deal with Microsoft to provide search services in 1998; Nasdaq-100 IPO in 1999.
A popular early alternative web directory to Yahoo! LookSmart is still operating today as a ‘premier search advertising network and management company’.
Can this type of bot really be hot? Who are we to judge?
Wired magazine release its HotBot search engine served by the Inktomi database and LookSmart and after that DMOZ directories.
Lycos acquired HotBot as part of its Wired magazine buy-out in 1998 and would eventually sell the hotbot.com domain name in 2016 for $155,000 to an anonymous buyer. Inktomi would be bought by Yahoo in 2003.
Hyperlink Search Engine or RankDex
Robin Li establishes RankDesk which uses backlinks to measure the quality and popularity of websites it is indexing. It uses the actual keywords in the anchor text of links pointing to a webpage to identify relevance.
RankDesk was also cited as an influence in Larry Page’s patent for PageRank.
Collaboration by Larry Page and Sergey Brin so named because it could count and analyse citation notation or backlinks pointing to a given site.
The anchor text and backlink data gathered for a given website or page would eventually be converted into a measure of importance by an algorithm called PageRank.
Ask Jeeves was founded by Garrett Gruener and David Warthen from Berkeley, California in the US.
Named after P. G. Wodehouse’s brilliant valet it was heavily marketed as an ‘answer engine’ and a more human way to search.
It would go public in 1999 but then be bought by InterActiveCorp in July 2005 with Jeeves retiring to become ask.com in 2006.
Asked to carry on, Jeeves made a comeback on the UK and Irish editions of the site in 2009.
IBM MedSpeak Radiology
IBM launches MedSpeak the first commercially available product capable of recognising continuous speech.
Based on a catalogue of the best Czech web pages to visit Ivo Lukačovič launches Seznam (or list).
Fuelled by the revenue from banner advertisements Seznam quickly added people search, a free email service and a news portal. In 2000, Lukačovič sold a 30% share to the Swedish portal, Spray International which became Lycos Europe.
Like PageRank, Seznam uses a 0-10 ranking scale for its websites called S-Rank.
Latest reports suggest that Google and Seznam share 50%-50% of desktop search in the Czech Republic, with Google dominating mobile searches. However, with new ventures such as a detailed wind map aimed at sailors and surfers called Windy it looks like Lukačovič and Seznam are going to continue to lead the way in terms of the range and innovation of services offered to users.
Arkady Volozh and Ilya Segalovich launch their search engine, Yandex or ‘Yet Another Indexer’ at the Softool exhibition in Moscow.
Yandex floated on the Nasdaq-100 in 2011 and is still Russia’s leading desktop search engine, just.
Google is rapidly eating into Yandex’s lead thanks in part to its domination of searches from mobile devices using its Android OS
Larry Page registers the domain name Google.com on September 15.
Gnuhoo or NewHoo or Open Directory Project or DMOZ
Created by Rich Skrenta as Gnuhoo then renamed NewHoo and when acquired by Netscape, the Open Directory Project also known as DMOZ (breathe).
It would become the largest internet directory and was run almost entirely by volunteer editors.
Officially closed in March 2017 but a version still exists http://dmoz-odp.org/.
Launched by Bill Gross, GoTo was the first company to allow advertisers to bid against each other auction-style to secure placement at the top of search results for specific search terms. Pay-Per-Click (PPC) or paid-for search advertising was born.
GoTo would become Overture Services in 2001 and form strategic partnerships with Microsoft and Yahoo! before acquiring Altavista and AlltheWeb.
It would eventually be taken over by one of its partners Yahoo! in 2003.
After the buzz created by BackRub Larry Page and Sergey Brin launch Google based on their unique PageRank technology.
In the words of Google themselves ‘PageRank works by counting the number and quality of links to a page to determine a rough estimate of how important the website is. The underlying assumption is that more important websites are likely to receive more links from other websites.’
The first Google index of the world wide web contained 25 million pages.
Microsoft launches MSN Search based on the Inktomi database which would be replaced by LookSmart and then AltaVista.
Based on a doctorate thesis by Tor Egge, AlltheWeb was completely customisable, had a bevy of advanced search options and featured search clustering. Bought by Overture in 2003.
Naver (from ‘Navigator’) was the first South Korean search portal to use an internally developed search engine.
While there is no doubt that today Naver is the predominant internet-based company in South Korea it’s more difficult to get an accurate figure of its pure search engine market share.
Currently, Dublin-based StatCounter suggests that Google has caught up considerably with just over 67% of the search market, while KoreanClick says that Naver has 74%.
Based on Robin Li’s pioneering RankDesk technology, Baidu was co-founded with Eric Xu.
From launch Baidu allowed users to bid for ad space and charged them for every click through.
According to StatCounter Baidu’s search market share in China today is 76.05% of an estimated internet-user population of 731 million.
Google launches its first advertising service based on a CPM or cost-per-mille model meaning that advertisers paid per thousand impressions of their ad.
Google would quickly move to a pay-per-click auction style model and go on to develop Quality Score to better determine the relevancy (and quality) of ads to search terms and their landing pages.
Naver – Comprehensive search
The first company to develop blended search results, Naver adds blogs, websites, images and businesses in its result pages.
Google’s index hits one billion unique pages.
Built by Matt Wells formerly of Infoseek, Gigablast is an open source search engine written in allowing for real-time indexing of websites.
Google Tool Bar — PageRank Opens to the public
Google launches its browser-based toolbar providing access to its search technology from any web page. Also included was a PageRank indicator.
Suddenly PageRank wasn’t just available to those in the know, anyone who had a website could see what theirs was and, perhaps more tellingly, what their competitors was. Websites now had a value placed on them, out of 10, by Google.
A major accelerant for the SEO industry as it is today, Google has spent the last five years talking down the importance of PageRank.
Google image search
Google launches a service allowing users to search the web for images. It would eventually enable users to search by submitting an image instead of keywords.
Microsoft Speech Recognition
Microsoft integrates speech recognition into their Office products for the first time.
Naver – Knowledge search
Naver becomes an early advocate of user-generated-content when it created its question and answer service.
Microsoft and Google Merger Rumours
In the fall of 2003 it is widely-reported in mainstream media that Microsoft and Google are holding merger talks.
Boston – Official Google Algorithm Update
Google’s first ‘official’ algorithm update.
Named Boston by Webmaster World users it signalled the introduction of the monthly algorithm and index refreshes which became affectionately referred to as ‘Google is dancing’ in the SEO industry.
A very useful phrase to explain an inexplicable drop in rankings to a vexed boss.
Cassandra & Dominic – Official Google Algorithm Updates
Cassandra was Google’s first ‘official’ algorithm update to focus on backlinks quality.
Dominic looked to attribute more weight to backlinks from reputable sites.
Both were named by Webmaster World users and were aimed at disrupting shady link building practises.
August 19: after a pre-IPO ‘road show’ described as unimpressive by financial analysts and weeks of the Nasdaq-100 trending downward, Google goes public at $85 a share.
After selling 22.5 million shares and raising $1.9 billion, Google shares end the day up 18% at $100.34.
After a successful IPO Baidu becomes the first Chinese company in the Nasdaq-100 stock index, its share price rising 354% on its first day of trading.
Google, Yahoo! and Microsoft introduce the ‘nofollow’ attribute to help try and counter the rise of link spam.
Windows Live Search
Microsoft launches it Windows Live Search to replace MSN Search. By 2007 it would just be called Live Search after merging with Microsoft adCenter.
Google’s first attempt at speech recognition was a telephone based directory service.
Created by digital entrepreneur Gabriel Weinberg DuckDuckGo is a hybrid search engine that compiles its results from many sources including Wikipedia, About, Bing, Yahoo! and its own bot, imaginatively titled DuckDuckBot.
DuckDuckGo differentiates itself by ‘being the search engine that doesn’t track you’, it does this by not storing cookies, or details of your User agent and IP address.
A very popular search engine among Tor Browser users as together they offer almost complete end-to-end anonymity online.
Google’s index reaches one trillion unique urls.
Live Search is re-branded as Bing.
Microsoft had also mulled using ‘Bang’ though thought better of it when they realised ‘I binged it’ was a little less open to wilful misinterpretation than ‘I banged it’.
In the same year, Microsoft and Yahoo! announce a 10-year deal to power Yahoo! by Bing.
Apple’s voice-based intelligent personal assistant Siri launches as an app for iOS. Its speech recognition engine is powered by Nuance Communications.
Caffeine – Google upgrades its search architecture
Caffeine was designed to help Google return results faster, to cope better with rapidly updated information and to incorporate the strides it had made in 2009 with real-time search.
Allows users to search major search engines anonymously by creating an instant Virtual Private Network connection to mask IP addresses.
Built by an ex-Google employee Brian Kennish.
Panda – Official Google Algorithm Update
Focused on penalising plagiarised (or scraped), thin, poorly written or software-created website content, user-generated spam comments and keyword stuffing.
The Panda update signalled the start of a new paradigm in Google’s policing of those not-altogether-public-spirited, SEO methods known as ‘black-hat’ in the industry.
Google, Yahoo! and Microsoft introduce schema, granular metadata to help website owners categorise and describe information better on their web pages for the benefit of bots.
Yandex would join the initiative later in the year.
Google launches its location-sensitive natural language personal assistant which fully integrates its latest version of Google Voice Search.
Penguin – Official Google Algorithm Update
Looked to penalise site owners who had acquired masses of backlinks from low quality or spammy websites and/or over optimised search term or keyword patterns in the anchor texts of said backlinks.
Hummingbird – Official Google Algorithm Update
Another update aimed at keyword stuffing and low quality or thin content.
Hummingbird improved Google’s ability to match query intent with search results by making it possible for a web page to rank for a search term or keyword even if it doesn’t contain an exact match.
Pigeon – Official Google Algorithm Update
This update merged Google’s local algorithm with its core algorithm aiming to provide more useful, relevant and accurate local search results.
Google Mobile Update
In late February Google announced it was expanding its ‘use of mobile-friendliness as a ranking signal.’
After weeks of Googling and debating ‘responsive versus adaptive’ design corporate and individual website owners alike huffed and puffed to transform their websites before the given April 21 deadline.
Industry insiders were quick to label it ‘mobilegeddon’ as a reaction to being given this very specific deadline by Google.
In March Naver appoints Han Seong-sook as its CEO — the first woman to lead a major Korean tech company.
Growing your business with seo...
Are you ready to see how we can help you grow your business in the organic search channel? Contact us today for a free quote...