Skip to content
Tutorials Center
My learning

Topics

Lesson 5 of 36 ยท How Search Engines Work

Crawling, Indexing and Ranking: How Search Engines Work

See how a search engine discovers pages, stores them in an index and ranks results, with a working mini crawler and inverted index you can run and change.

A search engine does three big jobs before it shows you a result: it crawls the web to discover pages, indexes what it found so that it can be searched quickly, and ranks the indexed pages when someone types a query. If your page fails at any of these steps, it will not appear. Knowing which step failed is the foundation of SEO troubleshooting.

This lesson explains each step in plain language and then lets you build tiny versions of a crawler and an index in Python so that the ideas are concrete.

Step 1: crawling

A crawler (also called a spider or bot) is a program that downloads pages and follows the links it finds. It keeps a queue of URLs to visit, called the crawl frontier. It starts from known pages, such as sitemaps and previously known URLs, fetches each page, extracts the links, and adds the new ones to the queue.

Important facts about crawling:

  • Crawlers discover pages mostly through links and sitemaps. A page with no links pointing to it (an "orphan") may never be found.
  • Crawlers have a limited crawl budget for each site: how many requests they are willing to make. Very large sites with many near-duplicate URLs can waste it.
  • Your robots.txt file tells crawlers which paths they should not request. It controls crawling, not indexing.
  • Crawlers identify themselves with a user-agent name, and good bots obey robots.txt.
  • Slow servers and error responses make crawlers visit less.

Try it: a mini crawler that obeys robots.txt

The program below crawls a made-up site stored in a dictionary (no network is used). It reads a robots.txt string with Python's urllib.robotparser, skips disallowed paths, follows links in breadth-first order and reports pages it could never reach.

Python Mini crawler with robots.txt

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Change a Disallow line or add a link to /hidden-landing and run it again. You will see why internal links matter for discovery and why robots.txt rules affect what gets requested.

Blocking a URL in robots.txt stops a polite crawler from fetching it, but the URL can still appear in results (without a description) if other sites link to it. To keep a page out of the index, use a noindex directive that the crawler is allowed to read. We cover this in Technical SEO.

Rendering

Many modern pages build their content with JavaScript. After downloading the HTML, a search engine may render the page, running scripts in a browser-like environment to see the final content. Rendering costs more than reading plain HTML, so it can happen later than the first fetch. Content that appears only after user actions (clicking a tab, scrolling forever) may be missed. For important text and links, make sure they are present in the HTML or rendered output without interaction.

Step 2: indexing

After a page is fetched (and rendered if needed), the engine analyzes it: language, topic, headings, links, images, structured data, and whether it duplicates another page. If it is considered worth keeping, it is stored in the index, a gigantic database built for fast lookup. Not everything crawled is indexed. Pages may be left out because they are duplicates, very low value, blocked by noindex, redirected to another URL or return errors.

The core data structure is an inverted index: instead of storing "page, then the words it contains", it stores "word, then the pages that contain it". Looking up a word is fast, and combining lists answers multi-word queries.

flowchart LR
    P["Fetched page"] --> T["Extract text, links, metadata"]
    T --> N["Normalize: lowercase, remove punctuation, stem words"]
    N --> I["Add to the inverted index<br/>word โ†’ list of pages"]
    T --> D{"Duplicate or low value?"}
    D -->|Yes| X["Left out or merged with a canonical page"]
    D -->|No| I

Indexing turns pages into searchable records and decides which URL represents duplicate content.

Try it: build an inverted index

This example builds a small inverted index from four tiny documents and answers queries by intersecting the word lists. Real engines add positions, many languages and billions of documents, but the structure is the same.

Python Inverted index search

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The query "pizza" returns an empty list because no document contains it. Searches with several words only return documents that contain all of them in this simple model; real engines also handle synonyms and partial matches.

Step 3: ranking

When a user searches, the engine finds the matching candidates in the index and ranks them. It looks at hundreds of signals, grouped roughly into relevance (does the page answer the query?), quality (is it reliable and well made?), usability (does it load fast, work on mobile, avoid intrusive pop-ups?), context (location, language, device, search history settings) and links from other sites. The next lessons, How Search Engines Rank Pages and Search Intent and SERP Features, examine these in depth.

Ranking is computed fresh for each query. The same page can rank first for one phrase and nowhere for another.

Where problems appear

Symptom Likely stage First thing to check
Page is not in the engine at all Crawling Is it linked internally? In the sitemap? Blocked in robots.txt?
Page was crawled but not indexed Indexing Duplicate? Thin? noindex tag? Canonical pointing elsewhere?
Indexed but ranks poorly Ranking Relevance to the query, quality, competition, links
Indexed, ranks, but few clicks Presentation Title, snippet, rich results

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the correct order of the three main jobs of a search engine?

    Pages must be discovered (crawled) and stored (indexed) before they can be ranked for a query.
  2. 2. What does an orphan page mean in crawling terms?

    Crawlers follow links, so a page nothing links to may never be discovered.
  3. 3. What does an inverted index map?

    An inverted index lists, for each word, the documents containing it, which makes lookups fast.
  4. 4. Which tool should keep a page out of the index?

    robots.txt controls crawling. noindex controls indexing, and the crawler must be able to fetch the page to see it.

Summary

  • Crawlers discover pages through links and sitemaps; robots.txt controls what they request; crawl budget is limited.
  • Rendering may happen after the first fetch, so important content should not depend on user actions.
  • Indexing stores selected pages in an inverted index; duplicates and low-value pages may be left out.
  • Ranking happens per query using relevance, quality, usability, context and links.
  • Diagnose problems stage by stage: crawl, index, rank, presentation.

Next: How Search Engines Rank Pages