Skip to content
Tutorials Center
My learning

Topics

Lesson 48 of 77 ยท Middleware, Signals and Internals

Sitemaps, RSS Feeds and robots.txt

Help search engines and readers find your content: build sitemaps with django.contrib.sitemaps, RSS and Atom feeds with the syndication framework, a robots.txt, and use contrib.sites for domains.

A good site is not only well written, it is discoverable. Search engines need a list of the pages you want indexed. Readers who follow blogs want a feed their reader can subscribe to. Crawlers need to know which parts of the site to leave alone. Django has built-in tools for all three, each only a few lines long:

Need Tool Output
"These are my pages" django.contrib.sitemaps /sitemap.xml
"These are my newest items" django.contrib.syndication RSS or Atom feed
"Please do not crawl these paths" A tiny view or template /robots.txt
"What is my domain name?" django.contrib.sites Site model used by the tools above

None of these need extra packages. They all build absolute URLs (https://example.com/posts/hello/), which is why they rely on the sites framework to know your domain. The examples below use a small Post model like the one in the blog project.

The sites framework: knowing your own domain

Feeds and sitemaps must contain full URLs, but a view cannot always know which domain it is serving (think of the same code deployed to staging and production). django.contrib.sites stores the domain in the database:

Python
# settings.py (read-only)
INSTALLED_APPS = [
    "django.contrib.sites",
    "django.contrib.sitemaps",
    "django.contrib.syndication",
    # ...
]
SITE_ID = 1

After migrate, there is one Site row, example.com. Change it to your real domain in the admin (Sites), or sitemaps will list example.com URLs, a mistake that happens surprisingly often. Without SITE_ID, Django can match the request's host instead (Site.objects.get_current(request)). You can also set protocol = "https" on a sitemap class so URLs use HTTPS.

Sitemaps

A sitemap is an XML file that lists your URLs with optional hints (last modification, how often it changes, relative priority). You describe each group of pages with a Sitemap subclass:

  • items() returns the objects (or names) to list.
  • location(item) returns the path; by default it calls item.get_absolute_url(), so give your models that method.
  • lastmod(item), changefreq, priority are optional hints. Accurate lastmod values help; made-up priority values are usually ignored by search engines.
  • limit (default 50,000 URLs per file, the protocol maximum) controls pagination for large sites.

Always order items() (order_by("pk") at least), because Django paginates the result.

Python A sitemap index with model and static pages, plus robots.txt

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check what the sitemap contains. The draft post is missing, the dates come from lastmod, and both sections use the real domain that was saved in the Site row. The sitemap index is how you organize big sites: one small file per section, so each stays below 50,000 URLs and you can see which section a search engine has trouble with.

For a single small sitemap you can skip the index and map path("sitemap.xml", sitemap, {"sitemaps": sitemaps}) with the name django.contrib.sitemaps.views.sitemap. Two practical tips:

  • Submit it. Give the URL to the search engines' webmaster tools (Google Search Console, Bing Webmaster Tools), and add a Sitemap: line to robots.txt as the example does. Django no longer pings search engines for you; that feature was removed because the engines stopped supporting it.
  • Cache it. Building a sitemap queries the database. Wrap the view in cache_page(86400) (see Caching) on large sites.

robots.txt

robots.txt is a plain text file at the root of the domain that tells well-behaved crawlers where not to go. It is a request, not a security measure: anything secret must be protected by authentication, not hidden in robots.txt. Django has no special framework for it, because it is just text. Serve it with a template or TemplateView:

Python
# urls.py (read-only)
from django.views.generic import TemplateView

urlpatterns = [
    path("robots.txt", TemplateView.as_view(template_name="robots.txt", content_type="text/plain")),
]
TEXT
# templates/robots.txt (read-only)
User-agent: *
Disallow: /admin/
Disallow: /accounts/
Allow: /

Sitemap: https://www.example.com/sitemap.xml

A typo here can remove your site from search results, so double-check the Disallow lines on staging before you copy them to production. To keep a single page out of search results, Disallow is the wrong tool (a blocked page can still be listed from links); add <meta name="robots" content="noindex"> or an X-Robots-Tag header instead.

RSS and Atom feeds

A feed is an XML document with your latest items, so a feed reader (or another program) can notice new content. Django's syndication framework needs a Feed subclass that works much like a sitemap:

  • title, link, description describe the feed itself (they can also be methods).
  • items() returns the objects to publish. Order them (newest first) and limit them: feed readers want the latest 10 to 50 entries, not your entire archive.
  • item_title, item_description, item_pubdate, item_link describe each entry; item_link defaults to get_absolute_url().
  • feed_type = Atom1Feed switches from RSS 2.0 to Atom.

An instance of the class is the view: path("feed/", LatestPosts()). Feeds can also take URL arguments: define get_object(request, category_slug) to build a per-category feed at feed/<slug:category_slug>/.

Python An RSS feed and an Atom feed from the same queryset

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Both feeds list the same two newest posts. The & in the third post title was escaped (&amp;) for you, since the framework produces valid XML. Notice the RSS <guid> and <pubDate> elements, which let readers recognize an item they have already seen: keep get_absolute_url() stable, because changing a URL makes readers think it is a new item.

To let browsers and readers discover the feed, advertise it in your base template's <head>:

HTML
<!-- read-only -->
<link rel="alternate" type="application/rss+xml" title="Example blog" href="{% url 'rss' %}">

Other things to know:

  • Per-item extras such as item_author_name, item_categories and item_enclosure_url (for podcasts) are optional methods on the same class.
  • Feeds are cacheable like any view, and readers poll them often, so put cache_page or an HTTP cache in front of busy feeds.
  • A feed of full post text needs item_description to return HTML; Django escapes it for XML. Many sites publish a summary instead to bring readers to the site.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why must you change the default Site row (example.com) after migrating?

    The sites framework supplies the domain used in absolute URLs in sitemaps and feeds. Leaving example.com puts wrong links in both.
  2. 2. What does a Sitemap subclass need at minimum?

    items() lists what to include, and each item's location comes from get_absolute_url() or location(item). Other attributes are hints.
  3. 3. Is robots.txt a good way to hide private pages?

    robots.txt is public and advisory. It can even advertise the paths you list. Use login checks for anything private.
  4. 4. How do you serve an Atom feed instead of RSS with Django's Feed class?

    Feed uses RSS 2.0 by default. Setting feed_type to django.utils.feedgenerator.Atom1Feed produces Atom.

Summary

  • django.contrib.sites stores your domain; set the real one before generating absolute URLs.
  • A Sitemap subclass lists URLs (items(), location(), lastmod()); use an index for large sites, order the items, exclude drafts, submit the URL to search engines and mention it in robots.txt.
  • robots.txt is a public, advisory text file served by a template or TemplateView.
  • A Feed subclass with items(), item_title, item_description and item_pubdate is itself the view; feed_type = Atom1Feed gives Atom, and a <link rel="alternate"> makes the feed discoverable.

Next: Useful contrib Apps: humanize, flatpages, redirects.