Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 39 ยท Best Practices

Performance and Memory in Notebooks

Find slow cells and memory hogs: time with %timeit, profile with cProfile, measure memory, free big variables, use efficient types, chunk files and cache expensive results.

Notebooks make it easy to load a big file and forget about it. Soon the kernel is using 12 GB, every cell takes minutes, and the laptop fan sounds like a jet. Good performance work in a notebook follows one rule: measure first. Guessing which part is slow wastes hours; a timer and a memory counter usually point straight at the cause.

The IPython magics (%timeit, %prun, %memit) only exist in a real kernel, so they appear as read-only code here. The runnable cells use the standard library equivalents (time, timeit, cProfile, tracemalloc), which work in any Python and are what the magics are built on.

Time a cell

Python
%%time
result = expensive_function(data)

%time runs the code once and reports wall time and CPU time. For small, fast operations use %timeit, which runs many loops and reports an average and spread:

Python
%timeit sum(range(1_000))
%%timeit
total = 0
for i in range(1_000):
    total += i

The standard library does the same thing:

Python Timing two ways to do the same job

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Exact numbers depend on the machine, but sum() is consistently much faster than the explicit loop: built-in code runs in C. Use the minimum of several repeats, since noise only makes timings slower, never faster.

Profile to find the hot spot

If a cell calls many functions, a profiler shows where the time goes:

Python
%prun -s cumulative run_pipeline(data)     # sorted by cumulative time
%lprun -f clean_rows clean_rows(data)      # line-by-line (needs line_profiler)
Python Profile with cProfile

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The report names slow_unique as the expensive function, because x not in out scans a list on every step. Replacing the list with a set makes it dramatically faster. Without the profile you might have optimized the parsing instead.

Make the data smaller

Most notebook memory problems come from data frames. Check how much they use:

Python
df.info(memory_usage="deep")
df.memory_usage(deep=True).sort_values(ascending=False).head()

Common wins:

  • Read only the columns you need: pd.read_csv(path, usecols=["id", "amount"]).
  • Use narrower types: int32 or float32 instead of int64/float64 when the range allows it.
  • Use category for repeated text such as country codes.
  • Parse dates once and keep them as datetimes, not strings.
  • Use Parquet instead of CSV: it is compact, typed and much faster to read.
Python
df["country"] = df["country"].astype("category")
df["quantity"] = pd.to_numeric(df["quantity"], downcast="integer")

Measure memory in plain Python

Python Why category-like encoding saves memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The coded version uses a small fraction of the memory. pandas' category dtype does the same thing: it stores integer codes plus a table of distinct values.

Free memory you no longer need

Variables stay alive until the kernel restarts or you delete them:

Python
del big_df
import gc; gc.collect()

Remember that outputs hold references too: _, __ and Out[n] keep the last results, and the history may keep big data frames alive. Run %reset -f to clear the namespace (a restart is cleaner), or %xdel name to remove a name and references to it. If memory keeps growing after you delete things, a restart is faster than a hunt.

Handle files bigger than memory

Process in chunks instead of loading all at once:

Python
totals = {}
for chunk in pd.read_csv("huge.csv", chunksize=500_000, usecols=["region", "amount"]):
    for region, amount in chunk.groupby("region")["amount"].sum().items():
        totals[region] = totals.get(region, 0) + amount

When a single machine is not enough, move to tools that do this for you: Polars or DuckDB (fast single-machine engines that can query files without loading them), Dask or Spark for clusters. The notebook becomes the controller and the engine does the heavy work.

Cache expensive results

Do not recompute a ten-minute step each time you restart. Save the result and load it if present:

Python A simple cache decorator

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In practice use joblib.Memory, functools.lru_cache for in-memory caches, or a Parquet file. A cache needs a rule for when it is stale: include the input parameters and a code version in the file name.

Parallelism, briefly

Vectorized NumPy and pandas operations are almost always faster than Python loops (df["a"] * df["b"] rather than apply). When a loop is unavoidable, multiprocessing, concurrent.futures or joblib.Parallel use several cores. Be careful with multiprocessing on Windows and macOS inside notebooks: functions must live in an importable module (see moving code into modules), a good reason to extract them.

A working routine

  1. Reproduce the slowness with a small sample.
  2. Time the cell; profile if it calls many functions.
  3. Fix the biggest cost only, and time again.
  4. Check memory with df.info; shrink columns and read only what you need.
  5. Cache the stable, expensive steps.
  6. Restart the kernel after heavy sessions.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does %timeit do?

    It repeats the statement to give a reliable timing, unlike a single run.
  2. 2. Which change usually saves the most memory for a column of repeated country names in pandas?

    Categories store small integer codes plus one copy of each distinct value.
  3. 3. Why profile before optimizing?

    Guesses are often wrong; a profile points to the real hot spot.
  4. 4. A CSV file is larger than RAM. What is a good first approach?

    Chunked processing or an out-of-core engine avoids loading everything at once.

Summary

  • Measure with %time, %timeit and a profiler before changing anything.
  • Shrink data: read fewer columns, use smaller types and category, prefer Parquet.
  • Delete big objects, restart the kernel after heavy work, and process huge files in chunks.
  • Cache expensive steps and prefer vectorized code over loops.

Next: Notebook Security