Skip to content
Tutorials Center
My learning

Topics

Lesson 20 of 36 ยท Technical SEO

robots.txt and XML Sitemaps

Learn how robots.txt controls crawling, how XML sitemaps help discovery, the difference between blocking and noindex, and how to parse both with Python.

Two small files do a lot of work in technical SEO. robots.txt tells crawlers which parts of your site they may request. An XML sitemap lists the URLs you want search engines to know about. They sound like opposites, but they cooperate: one narrows the crawl, the other points to what matters.

robots.txt basics

The file lives at the root of the host: https://example.com/robots.txt. Each host (and each subdomain) needs its own. It is made of groups of rules:

TEXT
User-agent: *
Disallow: /cart/
Disallow: /search
Allow: /search/help

User-agent: BadBot
Disallow: /

Sitemap: https://example.com/sitemap.xml
  • User-agent names the crawler the group applies to; * means all.
  • Disallow blocks paths that start with the given prefix; an empty value allows everything.
  • Allow makes an exception inside a blocked path. The most specific (longest) matching rule wins.
  • Sitemap is independent of groups and can appear several times.
  • * matches any characters and $ anchors the end of a URL in rules used by the major engines.

robots.txt is a request, not security. Polite crawlers obey it; bad ones ignore it, and the file itself is public. Never use it to hide private data. Protect that with authentication.

Blocking is not the same as noindex

This is the most common mistake in the whole topic.

  • Disallow stops crawling. A blocked URL can still be indexed (without a snippet) if other sites link to it.
  • noindex (a meta robots tag or X-Robots-Tag header) stops indexing, but the crawler must be allowed to fetch the page to see it.

So to remove a page from results, allow crawling and send noindex. If you block the URL, the engine never sees the tag.

flowchart LR
  A["Crawler wants URL"] --> B{"Allowed by robots.txt?"}
  B -- "No" --> C["Not fetched; may still be listed from links"]
  B -- "Yes" --> D["Fetch page"]
  D --> E{"noindex present?"}
  E -- "Yes" --> F["Dropped from index"]
  E -- "No" --> G["Can be indexed"]

The crawler checks robots.txt first, and only fetched pages can reveal a noindex directive.

What to block

Reasonable candidates are internal search results, endless filter combinations, cart and checkout steps, and admin areas. Do not block CSS and JavaScript files that pages need to render; search engines must fetch them to see the page the way users do.

Python Test rules with robotparser

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Note that Python's parser applies rules in order, so the Allow line is placed first here; major search engines pick the longest match regardless of order. Always test with the tools your search engine offers as well.

XML sitemaps

A sitemap is an XML file that lists canonical URLs, optionally with a last-modified date:

XML
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/running/shoes</loc>
    <lastmod>2025-03-14</lastmod>
  </url>
</urlset>

Guidelines:

  • Include only indexable, canonical, 200-status URLs. Redirects, noindex pages and duplicates send mixed signals.
  • A file may hold up to 50,000 URLs or 50 MB uncompressed. Larger sites use a sitemap index that lists several sitemap files.
  • lastmod is useful only if it is truthful: update it when the content meaningfully changes, not on every deploy.
  • Major engines ignore priority and changefreq; you can omit them.
  • Submit the sitemap in the search engine's webmaster tool and reference it in robots.txt.
  • Specialized sitemaps exist for images, videos and news.

A sitemap does not guarantee indexing. It is a hint that speeds discovery, especially for new sites, big sites and pages with few internal links.

Python Parse and lint a sitemap

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Sitemap index and compression

When a site is large, split it by type (sitemap-posts.xml, sitemap-products.xml) and list the files in an index. Splitting by section also helps you read indexing reports per section later in Search Console and Search Performance.

Generate a sitemap from a list

Remember to escape & as &amp; inside XML, as the code above does.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You want a page removed from search results. What should you do?

    The crawler must fetch the page to see noindex. Blocking it in robots.txt can leave the URL indexed without a snippet.
  2. 2. Which URLs belong in an XML sitemap?

    A sitemap should list the URLs you want indexed; redirects, duplicates and noindex pages contradict that.
  3. 3. Is robots.txt a security measure?

    The file is public and ignored by bad bots; use authentication for private content.
  4. 4. How many URLs can one sitemap file contain?

    The limit is 50,000 URLs or 50 MB uncompressed per file; use a sitemap index for more.

Summary

  • robots.txt controls crawling per host; it is public and advisory.
  • Disallow is not noindex: blocked pages can still be indexed from links.
  • Do not block resources needed to render pages.
  • Sitemaps list canonical, indexable URLs with honest lastmod values.
  • Use a sitemap index for large sites and escape special characters.

Next: Canonicals, Redirects and Duplicate Content