How search engines work

Here is an extract from Digital Marketing material explaining how search engines work. Search engines such as Google Search, Microsoft Bing, the Russian Yandex, Yahoo, Duck DuckGo, Baidu, AOL, Ask.com, Excite, Wolfram Alpha – the best computation search engine, Lycos for web hosting and emails.

• What is Wolfram Alpha?

Africa Digital Marketing Training – How search engines work

Hi there! Welcome to our video explaining how search engines work. We’ll give you the basics on how search engines find web pages, what they do with the pages they find, and how they decide what results to show. When you’re using a search engine to find the closest coffee shop, you’re probably not thinking about search engine technology. But later you might wonder, how did it do that? How did it sort through the entire Internet so quickly, and choose the results you saw on the page?

Each search engine uses their own software programs, but the way they work is pretty similar. They all perform three tasks: First, they examine content they learn about and have permission to see (that’s called crawling). Second, they categorise each piece of content (that’s called indexing). And third, they decide which content is most useful to searchers (that’s called ranking).

Let’s take a closer look at how these work. Search engines “crawl” the Internet to discover content, like web pages, images and videos. Each search engine uses computer programs called “bots” (short for robot), “crawlers” or “spiders” to make their way through the pages.

The bots hop from page to page by following links to other pages. These bots never stop; their sole purpose is to visit and revisit pages looking for new links and new content to include in the index. Indexing is the second part of the process. The index is a gigantic list of all the web pages and content found by the bots. The search engine uses this index as the source of information displayed on the search results pages.

But, not everything the bots find makes it into a search engine’s index.

For example, search engines may find multiple copies of the exact same piece of content, located on different websites.

How is that possible? Well, imagine you’re not searching for a coffee shop, but a coffeemaker. You might notice that the top-of-the-line CoffeeKing2000 has the same word-for-word description on the websites of many major retailers. The description might have been provided by the manufacturer… but now the search engine has decisions to make: which version to keep in the index? There’s no need for hundreds of duplicates, so it’s unlikely that every page will be added.

So if you own a website that’s selling coffeemakers, you’re likely better off writing your own description of the CoffeeKing2000.

Make sense? That covers crawling and indexing, which just leaves us with ranking. When you type in a search, the engine compares the words and phrases you use to its index, looking for matching results. Let’s say, for example, the search engine finds 230 million matching results. Now it’s time for the last part of the search engine’s task: ranking.

The way search engines rank pages is top secret—it’s their ‘special sauce.’ There are hundreds of ways search engines determine rank, including things like the words on the page, the number of other websites linking to it, and the freshness of the content.

But no matter what formula they use to determine rank, the goal remains the same: to try to connect the searcher with what they are looking for.

Say you’ve read about an Australian-style cappuccino called a flat white and you want to try it. If you search for “flat white coffee near me” the search engine will show you nearby shops selling the drink, because your search indicated your location. You might even see a map to help you find them. So, what have we learnt? Search engines are constantly working to scour the web for content, organise it and then display the most relevant results to searchers. Understanding this process will help you make your website the best it can be.