Skip to content
Tutorials Center
My learning

Topics

Lesson 64 of 77 ยท Deployment and Production

Performance and Scaling

Make Django fast and keep it fast: profile first, count queries, add indexes and caches, pool database connections, use a CDN and move slow work to the background.

Most slow Django sites are slow for boring reasons: too many database queries, a missing index, work repeated on every request that could have been done once. Very few need clever tricks. The skill is not knowing a hundred optimizations; it is finding the one thing that is actually slow and fixing that.

This lesson gives you a method (measure, fix the biggest cost, measure again), a tour of the layers where time goes (queries, caches, connections, the network, background work), and live examples that count queries and compare timings in your browser. Several tools mentioned here, such as the debug toolbar and connection pools, need a real project and database, so they appear as read-only snippets.

Measure first, then change one thing

Guessing is the most common performance mistake. Developers rewrite code that takes 2 ms and ignore the page that runs 300 queries. A reliable loop looks like this:

  1. Find a slow page or endpoint (user reports, logs, monitoring).
  2. Measure where the time goes: queries, Python code, template rendering, or an external call.
  3. Change one thing.
  4. Measure again, and keep the change only if the number improved.
flowchart LR
    A["Slow page"] --> B["Measure<br/>queries and time"]
    B --> C["Fix the biggest cost"]
    C --> D["Measure again"]
    D -->|"still slow"| B
    D -->|"fast enough"| E["Stop"]

Optimization is a loop. "Fast enough" is a valid end point.

Tools for measuring

  • django-debug-toolbar adds a panel to pages in development showing every SQL query, how long each took, duplicates, template rendering time and cache hits. It is a third-party package, so it is read-only here:

    Python
    # settings.py (development only)
    INSTALLED_APPS += ["debug_toolbar"]
    MIDDLEWARE.insert(0, "debug_toolbar.middleware.DebugToolbarMiddleware")
    INTERNAL_IPS = ["127.0.0.1"]
    
    # urls.py
    urlpatterns += [path("__debug__/", include("debug_toolbar.urls"))]
    
  • Query counting in tests. assertNumQueries and CaptureQueriesContext turn "this page does too many queries" into a failing test that stays fixed.

  • cProfile (in the standard library) shows which Python functions are called most and consume the most time.

  • Production monitoring. Slow query logs in your database, an APM service or tracing show what real users experience, which a laptop cannot.

The first live example uses a tiny profiler to show how call counts reveal a hotspot, and how timeit-style comparisons settle arguments with numbers.

Python Profiling a hotspot and comparing alternatives

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The profiler output is exact: normalize ran thousands of times for just five lookups, which points straight at the waste. The timing comparison prints only a yes or no, because exact times differ from machine to machine; what matters is the direction and the size of the difference.

Database queries: the biggest cost

In a typical Django app, the database accounts for most of the request time, and the most common problem is the N+1 query: one query to fetch a list, then one more per row to fetch something related. The full story is in ORM Performance; here is the short recap:

Problem Fix
Related object read in a loop (book.author) select_related("author") for foreign keys
Related list read in a loop (author.books.all()) prefetch_related("books")
Loading columns you never use only(), defer() or values()
len(qs) just to count qs.count(), or qs.exists() for a yes/no
Creating rows one by one bulk_create()
Filtering on a column with no index db_index=True or Meta.indexes

The best way to keep this under control is to make query counts visible. The next example adds a small middleware that records how many queries each request used in an X-Query-Count header, then compares a slow and a fast version of the same page.

Python Counting queries per request with middleware

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The slow page runs one query for the books plus one for each of the 180 books (181 in total); the fast page runs a single JOIN. With SQLite in memory each query is cheap, so the gap here is small. With a database on another machine, every query adds network time, and the difference becomes dramatic. That is why the count is the number to watch: it is stable and independent of your machine.

In a real project you would place this middleware in a module and list it by its dotted path, or use the debug toolbar. A test that says with self.assertNumQueries(2): around the page protects the fix forever.

Indexes and caches

Indexes

An index lets the database jump to matching rows instead of reading the whole table. Add one to columns you filter, sort or join on often. Django adds indexes for primary keys, foreign keys and unique=True columns automatically. For others:

Python
class Event(models.Model):
    kind = models.CharField(max_length=20, db_index=True)
    created = models.DateTimeField()

    class Meta:
        indexes = [models.Index(fields=["kind", "-created"])]

Indexes are not free: they take space and slow down writes slightly, so index what you query, not everything. You can ask the database how it plans to run a query with queryset.explain().

Caches

If a result is expensive and does not change on every request, store it. Django has a cache at each level, from fine-grained to coarse:

flowchart TB
    A["Browser cache<br/>(Cache-Control headers)"] --> B["CDN / reverse proxy"]
    B --> C["Per-view cache<br/>(cache_page)"]
    C --> D["Template fragment cache"]
    D --> E["Low-level cache API<br/>(cache.get_or_set)"]
    E --> F[("Database")]

A request is answered by the first layer that has the result. Each layer closer to the user saves more work.

The Caching lesson covers the API in depth. The next example uses the in-memory (locmem) backend to show the two things that matter most: a cached result costs zero queries, and the explain plan changes when an index exists.

Python Cache hits and index plans

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first request to each URL runs the queries; the second reads from the cache and runs none. Deleting the key forces a recompute, which is the simplest form of cache invalidation. Remember the hard part of caching is not storing data but deciding when it is stale. Cache things where a few seconds or minutes of staleness is acceptable, never data that must be exact (a bank balance), and never responses that differ per user unless the cache key includes the user.

Connections and the database server

Opening a database connection is slow, especially over a network with TLS. By default Django opens a new connection for every request and closes it afterwards. You have two options on PostgreSQL:

Python
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "app",
        "CONN_MAX_AGE": 60,          # reuse a connection for up to 60 seconds
        "CONN_HEALTH_CHECKS": True,  # drop dead connections before reuse
    }
}

Alternatively, Django 5.1 and newer can use a real connection pool through the psycopg (version 3) driver:

Python
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "app",
        "OPTIONS": {"pool": {"min_size": 2, "max_size": 10}},
    }
}

The pool keeps a small set of open connections and hands them out, which also protects the database from being flooded when traffic spikes. Do not set CONN_MAX_AGE and a pool together. Make sure your total connections (processes times pool size) stay below the database's limit; an external pooler such as PgBouncer is the usual answer when you run many web processes.

Scaling out

When one server is not enough, you add more copies of the web process. This works because a well-built Django app is stateless: nothing important lives in a process's memory or on its local disk.

  • Keep sessions in the database or a shared cache, not in per-process memory.
  • Keep uploaded files in object storage (S3 or similar), not on the web server's disk.
  • Use a shared cache (Redis or Memcached) instead of locmem, which is separate in every process. The locmem backend in the examples is for learning only.
  • Serve static files from a CDN (a network of caches close to your users) with long cache lifetimes and hashed file names, so a deploy changes the name when the content changes.
  • Send slow work to workers: emails, reports, image processing and calls to third-party APIs should not make a visitor wait. See Docker, Celery and CI.
  • Put the database on its own server, add read replicas for read-heavy workloads, and only then consider more complex designs.

Scale in this order: fix queries, add indexes, cache, use background jobs, add more web processes, and only then reach for exotic infrastructure. Each step is cheaper than the next, and the early steps often make the later ones unnecessary.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What should you do first when a Django page feels slow?

    Without measurement you may optimize something that takes almost no time. Measuring shows the biggest cost, which is the thing worth fixing.
  2. 2. A list page runs 1 query for 50 books and then 1 query per book to read its author. How can you reduce this to 1 query?

    select_related joins the foreign key in the same query, so the author data arrives with each book. (prefetch_related would work but uses a second query.)
  3. 3. Why is the number of queries a better thing to assert in tests than the elapsed time?

    A count is deterministic, so a test can fail reliably when someone reintroduces an N+1 problem. Timings depend on hardware and load.
  4. 4. Which setup lets you add more web servers safely?

    When state lives in shared services instead of one process or disk, any server can answer any request, so you can add or remove servers freely.
  5. 5. What is the main risk of caching a result?

    A cache trades freshness for speed. Choose timeouts and keys carefully, and invalidate when the underlying data changes.

Summary

  • Optimize by measuring: find the biggest cost, change one thing, measure again, and stop when it is fast enough.
  • Database queries usually dominate. Fix N+1 problems with select_related and prefetch_related, use count() and exists(), and add indexes to columns you filter on.
  • Make query counts visible with middleware, the debug toolbar or assertNumQueries; counts are stable where timings are not.
  • Caches work at many layers (browser, CDN, view, fragment, low-level API); a cache hit costs no queries, but you must plan for stale data.
  • Reuse database connections with CONN_MAX_AGE or a connection pool, and keep the total number of connections under the database limit.
  • Scale out by keeping the web tier stateless, sending slow work to background workers and serving static files from a CDN.

Next lesson: Project: Blog.