Who Invented The Search Engine? An Early History of Search Engines

1990

Archie

Arguably the first search engine, Archie (or Archive without the ‘v’ and not a comic book character) was built by Alan Emtage, Bill Heelan and J. Peter Deutsch while they were computer science students at McGill University in Montreal, Canada.

It started as a local network-based searchable database of file names of File Transfer Protocol (FTP) archives.

The first commercial version of Archie—a complete rewrite of the software—was made available in October of 1992 by Emtage and Deutsch who had by then founded Bunyip Information Systems.

 

1991

Veronica

Used for searching Gopher protocol servers, Veronica was built by Steven Foster and Fred Barrie while at the University of Nevada in the US. Gopher is a read-only network file system consisting of hierarchical hyperlinkable menu items (similar to a web page) and was invented by Mark McCahill at the University of Minnesota in the US.

Veronica provided keyword search of a database of menu items from files on Gopher servers. Named after a character, Veronica Lodge, created by US comic book publisher Archie Comics.

Jughead

Developed by Rhett Jones at the University of Utah in the US, Jughead also used the Gopher protocol but would search only a single server at a time.

And yes, it was also named after a character created by Archie Comics.

 

1992

The World Wide Web Virtual Library or VL

Built by the inventor of the world wide web and Londoner Tim Berners-Lee.

It was the first catalogue or directory of the world wide web and comprised of topic experts maintaining lists of links to quality resources.

You can still browse a version of The WWW Virtual Library today.

 

1993

World Wide Web Wanderer

Developed by Matthew Gray, it was the first bot ever built and was designed to measure the growth of the web by counting active web servers.

It created one of the first indexes called the Wandex and was eventually upgraded to include URLs.

W3 Catalog

Created by Oscar Nierstrasz at the University of Geneva, it mirrored web resources linked to by high-quality web directories, reformatted the content and provided front-end dynamic querying.

Aliweb

Dutch software engineer Martijn Koster’s early search engine did not use a bot instead relying on website owners to submit pages they wanted indexing.

Koster also created the Robots Exclusion Standard which allowed website owners to block access to bots at a domain or individual page level.

JumpStation

The first complete search engine to use bots to build and maintain an index made up of document titles and headings and offer a web form for users to search the index using keywords — how modern search engines work.

Created by Jonathan Fletcher from Scarborough, North Yorkshire while working at the University of Stirling in Scotland.

Eventually abandoned due to lack of investment and server space(!).

World Wide Web Worm

Developed by Oliver McBryan at the University of Colorado in the US, it was the first implementation of a search engine which used the anchor text in a backlink to characterise the page it points to.

It would eventually be cited as an influence in Larry Page’s patent for PageRank.

Back to top of page

1994

Jerry and David's Guide to the World Wide Web or Yahoo!

US-based Stanford University students David Filo and Jerry Yang start their own directory formed from their favourite web pages.

Sensibly re-named Yahoo! —after that race of brutish creatures from Swift’s Gulliver’s Travels— it quickly expanded and uniquely offered a human authored description with each url.

Today Yahoo! is still a thriving brand, powered by Microsoft’s Bing and owned by US multinational telecoms giant Verizon.

WebCrawler

Built by Brian Pinkerton WebCrawler was the first search engine to use bots to crawl and index entire web pages which has become the standard ever since. Users could search for any word on any webpage on the world wide web.

Bought by America Online in 1995 who would then sell WebCrawler to Excite in 1997. Eventually acquired by InfoSpace in 2001 when Excite (@Home) went bankrupt.

Today WebCrawler is a meta search engine blending results from Google, Yahoo! and ‘other popular search engines’.

Lycos

After starting as a research project by Michael Mauldin Lycos launched with a catalogue of 54,000 documents.

It provided results based on ranked relevance retrieval, prefix matching and word proximity.

Lycos became the fastest IPO from inception to offering in 1996. Bought by South Korea-based Daum Communications in 2004 who then sold Lycos to India-based Ybrant Digital in 2010.

Infoseek

Founded by Steve Kirsch, Infoseek started as a pay-for-use service but quickly became free post-launch. It uniquely allowed webmasters to submit a page for listing in real-time.

It became the default search provider for the then popular Netscape browser in 1995 and following this went public 1996.

Infoseek was bought by Walt Disney in 1998 and became Go.com. The technology behind Infoseek was then eventually bought by Inktomi in 2000.

Ex-Infoseek engineer Robin Li co-founded (and is current Chairman and CEO of) the China-based search engine Baidu.

Architext or Excite

Using statistical analysis of word relationships, Architext was a project by Graham Spencer, Joe Kraus, Mark VanHaren, Ryan McIntyre, Ben Lutch and Martin Reinfried all from Stanford University in the US.

Excite was formally launched in 1995 with an IPO quickly following in 1996. Bought by broadband provider @Home in 1999. Bankrupt in 2001 and acquired by InfoSpace.

 

1995

Altavista

Based on an initial idea by Paul Flaherty; Louis Monier wrote the web crawler and Michael Burrows the indexer.

It was the first search engine to allow natural language queries, had unlimited bandwidth and allowed users to add or delete a url within 24 hours.

Altavista was bought by Overture in 2003 which was then bought by Yahoo! in the same year.

Homebase or LookSmart

Founded in Melbourne, Australia, Homebase was majority owned by Reader’s Digest but was bought back from them by original founders Evan Thornley and Tracey Ellery; re-located to California and re-branded as LookSmart; signed deal with Microsoft to provide search services in 1998; Nasdaq-100 IPO in 1999.

A popular early alternative web directory to Yahoo! LookSmart is still operating today as a ‘premier search advertising network and management company’.

Back to top of page

1996

HotBot

Can this type of bot really be hot? Who are we to judge?

Wired magazine release its HotBot search engine served by the Inktomi database and LookSmart and after that DMOZ directories.

Lycos acquired HotBot as part of its Wired magazine buy-out in 1998 and would eventually sell the hotbot.com domain name in 2016 for $155,000 to an anonymous buyer. Inktomi would be bought by Yahoo in 2003.

Hyperlink Search Engine or RankDex

Robin Li establishes RankDesk which uses backlinks to measure the quality and popularity of websites it is indexing. It uses the actual keywords in the anchor text of links pointing to a webpage to identify relevance.

RankDesk was also cited as an influence in Larry Page’s patent for PageRank.

BackRub

Collaboration by Larry Page and Sergey Brin so named because it could count and analyse citation notation or backlinks pointing to a given site.

The anchor text and backlink data gathered for a given website or page would eventually be converted into a measure of importance by an algorithm called PageRank.

Ask Jeeves

Ask Jeeves was founded by Garrett Gruener and David Warthen from Berkeley, California in the US.

Named after P. G. Wodehouse’s brilliant valet it was heavily marketed as an ‘answer engine’ and a more human way to search.

It would go public in 1999 but then be bought by InterActiveCorp in July 2005 with Jeeves retiring to become ask.com in 2006.

Asked to carry on, Jeeves made a comeback on the UK and Irish editions of the site in 2009.

IBM MedSpeak Radiology

IBM launches MedSpeak the first commercially available product capable of recognising continuous speech.

Seznam

Based on a catalogue of the best Czech web pages to visit Ivo Lukačovič launches Seznam (or list).

Fuelled by the revenue from banner advertisements Seznam quickly added people search, a free email service and a news portal. In 2000, Lukačovič sold a 30% share to the Swedish portal, Spray International which became Lycos Europe.

Like PageRank, Seznam uses a 0-10 ranking scale for its websites called S-Rank.

Latest reports suggest that Google and Seznam share 50%-50% of desktop search in the Czech Republic, with Google dominating mobile searches. However, with new ventures such as a detailed wind map aimed at sailors and surfers called Windy it looks like Lukačovič and Seznam are going to continue to lead the way in terms of the range and innovation of services offered to users.

Back to top of page

1997

Yandex

Arkady Volozh and Ilya Segalovich launch their search engine, Yandex or ‘Yet Another Indexer’ at the Softool exhibition in Moscow.

Yandex floated on the Nasdaq-100 in 2011 and is still Russia’s leading desktop search engine, just.

Google is rapidly eating into Yandex’s lead thanks in part to its domination of searches from mobile devices using its Android OS

Google.com

Larry Page registers the domain name Google.com on September 15.

Back to top of page

1998

Gnuhoo or NewHoo or Open Directory Project or DMOZ

Created by Rich Skrenta as Gnuhoo then renamed NewHoo and when acquired by Netscape, the Open Directory Project also known as DMOZ (breathe).

It would become the largest internet directory and was run almost entirely by volunteer editors.

Officially closed in March 2017 but a version still exists http://dmoz-odp.org/.

GoTo.com

Launched by Bill Gross, GoTo was the first company to allow advertisers to bid against each other auction-style to secure placement at the top of search results for specific search terms. Pay-Per-Click (PPC) or paid-for search advertising was born.

GoTo would become Overture Services in 2001 and form strategic partnerships with Microsoft and Yahoo! before acquiring Altavista and AlltheWeb.

It would eventually be taken over by one of its partners Yahoo! in 2003.

Google

After the buzz created by BackRub Larry Page and Sergey Brin launch Google based on their unique PageRank technology.

In the words of Google themselves ‘PageRank works by counting the number and quality of links to a page to determine a rough estimate of how important the website is. The underlying assumption is that more important websites are likely to receive more links from other websites.’

The first Google index of the world wide web contained 25 million pages.

MSN Search

Microsoft launches MSN Search based on the Inktomi database which would be replaced by LookSmart and then AltaVista.

Back to top of page

1999

AlltheWeb

Based on a doctorate thesis by Tor Egge, AlltheWeb was completely customisable, had a bevy of advanced search options and featured search clustering. Bought by Overture in 2003.

Naver

Naver (from ‘Navigator’) was the first South Korean search portal to use an internally developed search engine.

While there is no doubt that today Naver is the predominant internet-based company in South Korea it’s more difficult to get an accurate figure of its pure search engine market share.

Currently, Dublin-based StatCounter suggests that Google has caught up considerably with just over 67% of the search market, while KoreanClick says that Naver has 74%.

Back to top of page

2000

Baidu

Based on Robin Li’s pioneering RankDesk technology, Baidu was co-founded with Eric Xu.

From launch Baidu allowed users to bid for ad space and charged them for every click through.

According to StatCounter Baidu’s search market share in China today is 76.05% of an estimated internet-user population of 731 million.

Google AdWords

Google launches its first advertising service based on a CPM or cost-per-mille model meaning that advertisers paid per thousand impressions of their ad.

Google would quickly move to a pay-per-click auction style model and go on to develop Quality Score to better determine the relevancy (and quality) of ads to search terms and their landing pages.

Naver – Comprehensive search

The first company to develop blended search results, Naver adds blogs, websites, images and businesses in its result pages.

Google index

Google’s index hits one billion unique pages.

Gigablast

Built by Matt Wells formerly of Infoseek, Gigablast is an open source search engine written in allowing for real-time indexing of websites.

Google Tool Bar — PageRank Opens to the public

Google launches its browser-based toolbar providing access to its search technology from any web page. Also included was a PageRank indicator.

Suddenly PageRank wasn’t just available to those in the know, anyone who had a website could see what theirs was and, perhaps more tellingly, what their competitors was.  Websites now had a value placed on them, out of 10, by Google.

A major accelerant for the SEO industry as it is today, Google has spent the last five years talking down the importance of PageRank.

Back to top of page

2001

Google image search

Google launches a service allowing users to search the web for images. It would eventually enable users to search by submitting an image instead of keywords.

Back to top of page

2002

Microsoft Speech Recognition

Microsoft integrates speech recognition into their Office products for the first time.

Naver – Knowledge search

Naver becomes an early advocate of user-generated-content when it created its question and answer service.

Back to top of page

2003

Microsoft and Google Merger Rumours

In the fall of 2003 it is widely-reported in mainstream media that Microsoft and Google are holding merger talks.

Boston – Official Google Algorithm Update

Google’s first ‘official’ algorithm update.

Named Boston by Webmaster World users it signalled the introduction of the monthly algorithm and index refreshes which became affectionately referred to as ‘Google is dancing’ in the SEO industry.

A very useful phrase to explain an inexplicable drop in rankings to a vexed boss.

Cassandra & Dominic – Official Google Algorithm Updates

Cassandra was Google’s first ‘official’ algorithm update to focus on backlinks quality.

Dominic looked to attribute more weight to backlinks from reputable sites.

Both were named by Webmaster World users and were aimed at disrupting shady link building practises.

Back to top of page

2004

Google IPO

August 19: after a pre-IPO ‘road show’ described as unimpressive by financial analysts and weeks of the Nasdaq-100 trending downward, Google goes public at $85 a share.

After selling 22.5 million shares and raising $1.9 billion, Google shares end the day up 18% at $100.34.

Back to top of page

2005

Baidu IPO

After a successful IPO Baidu becomes the first Chinese company in the Nasdaq-100 stock index, its share price rising 354% on its first day of trading.

Nofollow attribute

Google, Yahoo! and Microsoft introduce the ‘nofollow’ attribute to help try and counter the rise of link spam.

 

2006

Windows Live Search

Microsoft launches it Windows Live Search to replace MSN Search. By 2007 it would just be called Live Search after merging with Microsoft adCenter.

Back to top of page

2007

GOOG-411

Google’s first attempt at speech recognition was a telephone based directory service.

Back to top of page

2008

DuckDuckGo

Created by digital entrepreneur Gabriel Weinberg DuckDuckGo is a hybrid search engine that compiles its results from many sources including Wikipedia, About, Bing, Yahoo! and its own bot, imaginatively titled DuckDuckBot.

DuckDuckGo differentiates itself by ‘being the search engine that doesn’t track you’, it does this by not storing cookies, or details of your User agent and IP address.

A very popular search engine among Tor Browser users as together they offer almost complete end-to-end anonymity online.

Google index

Google’s index reaches one trillion unique urls.

Back to top of page

2009

Bing

Live Search is re-branded as Bing.

Microsoft had also mulled using ‘Bang’ though thought better of it when they realised ‘I binged it’ was a little less open to wilful misinterpretation than ‘I banged it’.

In the same year, Microsoft and Yahoo! announce a 10-year deal to power Yahoo! by Bing.

Apple Siri

Apple’s voice-based intelligent personal assistant Siri launches as an app for iOS. Its speech recognition engine is powered by Nuance Communications.

Back to top of page

2010

Caffeine – Google upgrades its search architecture

Caffeine was designed to help Google return results faster, to cope better with rapidly updated information and to incorporate the strides it had made in 2009 with real-time search.

Back to top of page

2011

Disconnect Search

Allows users to search major search engines anonymously by creating an instant Virtual Private Network connection to mask IP addresses.

Built by an ex-Google employee Brian Kennish.

Panda – Official Google Algorithm Update

Focused on penalising plagiarised (or scraped), thin, poorly written or software-created website content, user-generated spam comments and keyword stuffing.

The Panda update signalled the start of a new paradigm in Google’s policing of those not-altogether-public-spirited, SEO methods known as ‘black-hat’ in the industry.

Schema

Google, Yahoo! and Microsoft introduce schema, granular metadata to help website owners categorise and describe information better on their web pages for the benefit of bots.

Yandex would join the initiative later in the year.

Back to top of page

2012

Google Now

Google launches its location-sensitive natural language personal assistant which fully integrates its latest version of Google Voice Search.

Penguin – Official Google Algorithm Update

Looked to penalise site owners who had acquired masses of backlinks from low quality or spammy websites and/or over optimised search term or keyword patterns in the anchor texts of said backlinks.

Back to top of page

2013

Hummingbird – Official Google Algorithm Update

Another update aimed at keyword stuffing and low quality or thin content.

Hummingbird improved Google’s ability to match query intent with search results by making it possible for a web page to rank for a search term or keyword even if it doesn’t contain an exact match.

 

2014

Pigeon – Official Google Algorithm Update

This update merged Google’s local algorithm with its core algorithm aiming to provide more useful, relevant and accurate local search results.

Back to top of page

2015

Google Mobile Update

In late February Google announced it was expanding its ‘use of mobile-friendliness as a ranking signal.’

After weeks of Googling and debating ‘responsive versus adaptive’ design corporate and individual website owners alike huffed and puffed to transform their websites before the given April 21 deadline.

Industry insiders were quick to label it ‘mobilegeddon’ as a reaction to being given this very specific deadline by Google.

Back to top of page

2017

Naver appoints first female CEO

In March Naver appoints Han Seong-sook as its CEO — the first woman to lead a major Korean tech company.