Skip to content
Tutorials Center
My learning

Topics

Lesson 41 of 57 ยท Saving and Production

Plotting Large Data Fast

Make slow Matplotlib charts fast: aggregate with hexbin and hist2d, downsample, rasterize dense layers, set path chunking, close figures and profile what is slow.

Matplotlib is happy to draw a few thousand points, and it copes with a few hundred thousand. Somewhere past a million, charts become slow, vector files become huge, and a loop that makes many figures can eat all your memory. None of this is a bug. Every point is an object that has to be turned into pixels, and one chart can only show a few hundred thousand pixels anyway.

The fix is almost never "buy a faster computer". It is to ask what the reader can actually see and draw only that. This lesson covers the main tools in the order you should try them: aggregate the data, reduce it, rasterize what is dense, tune the renderer, and release figures you no longer need. It builds on scatter plots, histograms and saving figures.

Draw less, show the same

Aggregate Bin points into counts with hexbin or hist2d.
Reduce Downsample or decimate before plotting.
Rasterize Turn dense layers into pixels inside vector files.
Clean up Close figures you no longer need.

Measure first, then pick the cheapest fix.

1 / 2

Where the time goes

Three things cost time in a plot: building the artists (Python objects), drawing them (the renderer), and, if you save a vector file, writing every one of them to disk. A scatter of a million points makes one collection, but the renderer still has to place a million markers. A plot line with a million vertices is a single artist, but the renderer has to stroke a path with a million segments.

Before you optimize, measure. The standard library's time.perf_counter is enough: draw the figure and force a render with fig.canvas.draw(). Without that call, most work is deferred until the figure is saved or shown, and your timer would only measure artist creation.

Python Timing a draw honestly

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The numbers you get depend on your machine (and on the browser, here), so the example prints only a comparison. On your own computer, print elapsed and watch how it grows with n. Also notice plt.close(fig) inside the function. We come back to it at the end of the lesson.

Aggregate instead of plotting every point

When a scatter plot is one solid blob, the individual points carry no information. What the reader wants is where the data is dense. Aggregation answers that directly: split the plane into cells, count the points in each cell, and draw one colored cell per bin. The cost depends on the number of bins, not the number of points.

Matplotlib has two tools for this. ax.hist2d(x, y, bins=...) uses a square grid. ax.hexbin(x, y, gridsize=...) uses hexagons, which look smoother and have no directional bias. Both return a mappable that you can give to fig.colorbar. Use bins="log" (hexbin) or norm=LogNorm() (hist2d) when a few cells have far more points than the rest, as the lesson on colormaps explains.

Python Half a million points as a density map

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The chart shows two overlapping clouds with a clear dense core in each, which a plain scatter would have painted as an unreadable smear. The hexagon count printed at the end is the real amount of work the renderer did.

Reduce the data before plotting

Sometimes you do want a line, not a density map. A time series with five million samples drawn across a 1,000-pixel-wide axis puts thousands of samples into each pixel column. Plotting all of them is wasted effort. There are three common reductions.

  • Decimation: keep every k-th sample, x[::k]. It is fast and simple, but it can skip spikes and create false patterns (aliasing).
  • Random sampling: rng.choice(len(x), size=m, replace=False), good for scatter plots where order does not matter.
  • Min/max envelope: split the data into blocks and keep each block's minimum and maximum. The result has the same visible shape as the full data and never hides a spike. It is the most faithful reduction for signals.
Python Min/max envelope keeps the spikes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The top panel reduces the data to 1,000 samples, and the spike has vanished, as the printed maxima show. The bottom panel uses the same number of blocks, yet the spike is clearly visible. The shape you see in the bottom chart is the shape of all two million samples.

With pandas you can build the same reduction with resample or rolling().agg(["min", "max"]). Pick the reduction that matches the question: means for trends, min/max for spikes.

Rasterize dense layers in vector files

A PNG has a fixed number of pixels, so its size does not depend on how many points you drew. An SVG or PDF stores every object, so 200,000 scatter points become 200,000 entries and a file that is slow to open. The cure is rasterized=True. It tells Matplotlib to draw that one artist as a bitmap and embed it in the vector file, while the axes, text and tick labels stay sharp vector shapes.

You can pass rasterized=True to plot, scatter, pcolormesh, fill_between and most other drawing calls, or set it afterwards with artist.set_rasterized(True). Remember that the resolution of the embedded bitmap is the dpi you give to savefig.

Python Same chart, much smaller SVG

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The visible chart is identical in both versions, but the vector file shrinks by a large factor, which is the point of the option. The exact byte counts depend on the library version, so the example prints only comparisons.

Tune the renderer

Two smaller tools help with long lines.

Path simplification. For a line with many points, Matplotlib merges nearly collinear segments before drawing. The setting path.simplify is on by default and path.simplify_threshold (default 0.111 pixels) controls how aggressive it is. It rarely needs changes, but it explains why a huge line plot can be quicker than you expect. It does not apply to markers or to lines with many sharp turns.

Chunking. A single path with hundreds of thousands of vertices can make the Agg renderer fail with an OverflowError: Exceeded cell block limit. Setting plt.rcParams["agg.path.chunksize"] = 10000 makes it draw the path in pieces and avoids the error. Set it before you draw.

Other cheap wins:

  • Use a smaller figsize and dpi while experimenting; pixels cost time.
  • Avoid alpha on huge scatters; blending is slower than opaque drawing. Use a smaller s instead.
  • Use marker="," (a single pixel) or ax.plot(..., ",") for extremely dense clouds.
  • Call ax.set_autoscale_on(False) after setting limits if you add many artists in a loop, so Matplotlib does not recompute the limits each time.

Blitting for interactive and animated charts

Animations and interactive widgets redraw the same figure again and again. Most of the figure does not change between frames, so redrawing everything is wasteful. Blitting saves a bitmap of the static background once, then for each frame it restores that bitmap and draws only the changing artists marked with animated=True. Frame rates can rise from a few per second to dozens.

The function-based animation class takes blit=True and does this for you, and you can also do it by hand with fig.canvas.copy_from_bbox, restore_region and blit. It needs an interactive backend, so here it is read-only:

Python
fig, ax = plt.subplots()
(line,) = ax.plot(x, y, animated=True)      # excluded from normal draws
fig.canvas.draw()                           # draw the static parts
background = fig.canvas.copy_from_bbox(ax.bbox)

for frame in range(100):
    fig.canvas.restore_region(background)   # put the saved background back
    line.set_ydata(new_y(frame))
    ax.draw_artist(line)                    # draw only the moving line
    fig.canvas.blit(ax.bbox)                # push that region to the screen
    fig.canvas.flush_events()

You will see this in a wider setting in Animation Concepts.

Close figures to free memory

plt.subplots() and plt.figure() register every figure with pyplot, which keeps it alive until you close it. A loop that makes a thousand charts without closing them holds a thousand figures in memory, and Matplotlib warns once there are more than 20 open (RuntimeWarning: More than 20 figures have been opened). On a server, this is the classic memory leak.

The rule is simple: if you saved the figure, or no longer need it, call plt.close(fig). plt.close("all") closes every figure. Alternatively, build figures with matplotlib.figure.Figure() directly, which pyplot does not track, and let Python garbage-collect them.

Python Counting open figures

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Profiling a slow plot

When a chart is slow and you do not know why, profile it instead of guessing. Python's cProfile shows which functions take the time:

Python
import cProfile, pstats

profiler = cProfile.Profile()
profiler.enable()
fig.canvas.draw()                 # the part you want to measure
profiler.disable()
pstats.Stats(profiler).sort_stats("cumulative").print_stats(10)

Read the top of the list. If most time sits in functions with draw in their name, you are renderer-bound: reduce or aggregate the data. If it sits in your own data preparation (a pandas groupby, a Python loop over points), the fix is outside Matplotlib. In a notebook, %timeit and %prun do the same job.

A quick checklist for a slow chart:

  1. How many points does each artist have? Is that more than the pixels on the axes?
  2. Could hexbin, hist2d or an envelope show the same message?
  3. Are you saving SVG or PDF with a dense layer? Add rasterized=True.
  4. Is a loop creating figures without closing them?
  5. Is alpha, a huge figsize, or a very high dpi adding cost without improving the picture?

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A scatter plot of 3 million points is one solid blob. What is the best fix?

    Aggregation draws one cell per bin instead of one marker per point, which is faster and also shows the density that the blob hides. A bigger figure is slower, and SVG makes the file larger.
  2. 2. What does rasterized=True do when you save an SVG?

    Only the marked artist becomes a bitmap (at the savefig dpi). Axes, labels and the other artists stay as vector shapes, so the file is much smaller.
  3. 3. Why is a min/max envelope better than plotting every 2000th sample of a long signal?

    Each block contributes both its lowest and highest value, so a single-sample spike still appears. Decimation keeps only a sample per block and may miss it.
  4. 4. Which call releases a figure from pyplot's memory?

    pyplot keeps a reference to every figure it created until plt.close is called. clear() only empties the figure, and show() does not free it.

Summary

  • Charts are slow when artists outnumber what the reader can see; the cure is to draw less, not to draw faster.
  • Measure with fig.canvas.draw() and time.perf_counter, or cProfile, before changing anything.
  • hexbin and hist2d replace millions of points with a few thousand cells and show density clearly.
  • Reduce long signals with a min/max envelope (keeps spikes), decimation (fast, may alias) or random sampling.
  • rasterized=True embeds dense layers as bitmaps in SVG and PDF, keeping files small and text sharp.
  • agg.path.chunksize fixes "cell block limit" errors for huge paths; blitting speeds up interactive redraws.
  • Close figures with plt.close(fig) in loops and on servers to avoid memory leaks.

Next, turn one-off charts into repeatable pipelines in Reproducible and Automated Charts.