Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 31 ยท Production and Performance

Testing Charts and Common Pitfalls

Test chart specs instead of pixels, snapshot figure JSON, and avoid truncated bars, dual axes, overplotting and version drift in Plotly, Bokeh and Altair.

Charts are code, and like all code they break. A refactor reorders the columns and the bars change places. A new data load contains a missing value and one line quietly stops. A library upgrade changes a default and the colors shift. Nobody notices until a manager asks why the report says something different from last month.

Testing a chart sounds hard because the output is a picture. The trick is to not test the picture. Every one of these libraries builds a plain description of the chart first (a Plotly figure dict, a Vega-Lite spec, a Bokeh document), and that description is ordinary data you can assert on. This lesson shows how to test it, then goes through the design mistakes that most often make a chart lie. The libraries need Python and a browser display, so the Plotly, Bokeh and Altair snippets are read-only; the runnable examples are stand-ins that apply the same tests and checks to plain Python dictionaries shaped like a Plotly figure. For how those dictionaries are built, see Plotly Figures: Data, Layout and Traces and Reading and Writing Vega-Lite JSON.

Test the spec, not the pixels

Pixel comparisons are fragile. A different operating system renders fonts a little differently, a new browser version shifts anti-aliasing, and tests fail for reasons that have nothing to do with your code. A spec test avoids all of that, because it checks facts such as "there are three bars", "the y-axis has a title" and "the values equal the input".

Python
# Plotly: the figure is a dict-like object
fig = build_sales_chart(rows)
spec = fig.to_dict()
assert spec["data"][0]["type"] == "bar"
assert list(spec["data"][0]["y"]) == [120.0, 150.0, 190.0]
assert fig.layout.yaxis.title.text == "Sales (USD)"

# Altair: to_dict() also validates against the Vega-Lite schema
spec = build_sales_chart(rows).to_dict()
assert spec["mark"]["type"] == "bar"
assert spec["encoding"]["y"]["field"] == "sales"

Read-only. Altair raises an error for an invalid spec when you call to_dict(), which is itself a useful test. For Bokeh, inspect the figure's renderers and data sources, or its JSON through bokeh.embed.json_item.

flowchart LR
    A["Data + chart code"] --> B["Spec<br/>dict or JSON"]
    B --> C["Assertions<br/>facts about the chart"]
    B --> D["Snapshot<br/>compare with saved JSON"]
    B --> E["Browser draws pixels<br/>not tested here"]

Tests go on the spec, one step before the browser.

What to assert

Pick facts that would embarrass you if they were wrong:

  • Data: the plotted values equal the source values (an easy check, and it catches reordering and dropped rows). The number of marks matches the number of rows.
  • Completeness: no missing values where there should be none, and x and y have the same length.
  • Labels: the title, axis titles and units exist and are not empty.
  • Scale rules: bars start at zero, the date axis is a real date axis, a shared scale is really shared.
  • Mapping: the right column drives color, and the color mapping is stable across runs.

The stand-in below does this for a bar chart built from rows. Read the function build_figure as "the code under test", and the list of check calls as the test file.

Python Assertions and a snapshot on a figure dict

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

All seven checks pass for the correct data. The second half shows a snapshot test: you save the full spec as sorted, indented JSON in a file, and the test compares the current spec with it. When someone drops a row, the diff points at exactly the lines that changed (the missing month and its value), which is far more useful than "image differs by 0.3%". Snapshot tests have two rules:

  1. Make the snapshot stable. Sort keys, round floats to a sensible number of digits, and do not include things that change on every run, such as random IDs or timestamps. Plotly figures built with fig.to_dict() are stable; Bokeh documents include generated object IDs, so you may need to strip those before comparing.
  2. Review changes, don't just accept them. A snapshot is a record of what you believed was right. When it changes, read the diff the same way you read a code review.

Snapshots and assertions complement each other: assertions state why the spec is right, and the snapshot catches everything you did not think to assert. If you also want some picture-level safety, keep one or two end-to-end image checks, but treat them as a smoke test, not the main defense.

Pitfalls that make a chart lie

The next group of mistakes passes every technical test and still misleads. Several can be caught automatically; the rest need a human.

Truncated axes

A bar's length is the value. If the axis starts at 97 instead of 0, a bar for 98 and a bar for 100 look like a 1-to-3 ratio when the real difference is 2%. Bar and area charts must start at zero. For a line or a dot plot, a non-zero baseline is acceptable when the point is the shape of change, as long as the axis is clearly labeled. If the range matters, show it: a break marker, a note, or a different chart type.

Dual axes

A chart with two y-axes lets you draw two lines of different units on one picture, and it invites the author to choose the two scales until the lines look correlated. Change either range and the relationship changes. Prefer two stacked charts that share an x-axis, or index both series to 100 at the start so they can share one axis. If you must use two axes, color each axis to match its line and say why.

Overplotting

When thousands of points land in the same place, a scatter plot shows one blob and hides how many points are in it. Reduce the marker size, add transparency (alpha), jitter discrete values slightly, or bin the points as shown in Large Datasets and Performance.

More traps worth a checklist

  • Pie charts with many slices: people compare angles poorly; use a sorted bar chart beyond three or four slices.
  • Area encoded as radius: if bubbles are scaled by radius, a value twice as big looks four times as big. Scale by area (Plotly's sizemode="area", which is its default for px bubbles, and Altair's size scale do this by default).
  • Unsorted categories: sort bars by value unless the categories have a natural order such as months.
  • Rainbow scales and unlabeled color: see Color, Labels and Accessibility.
  • Cherry-picked time windows: a trend that looks steep over three months may be flat over three years; choose the window for the question, not the story.
  • 3D effects: perspective distorts lengths with no benefit.

The next stand-in turns three of these into automatic checks on a figure dict, which makes them cheap to run on every build. The last lines show a fourth kind of problem, version drift.

Python Automatic checks for common pitfalls

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The first check flags the bar chart whose axis starts at 97. The second finds the trace attached to a secondary axis. The third shows that 5,000 points occupy just 70 distinct positions, with up to 72 points stacked in one spot, which is exactly what a flat scatter plot would hide. These checks are blunt: a legitimate truncated line chart should not be flagged, so in a real project you would apply each rule only to the chart types it concerns, and let a person decide the borderline cases.

Library version drift

Charting libraries move quickly, and charts in old reports must keep working. Some examples of what has changed between major versions: Altair 5 replaced add_selection with add_params, Bokeh 3 removed plot_width and plot_height in favor of width and height, and Plotly 6 removed some long-deprecated attributes. Smaller changes, such as default colors, fonts or hover text, are harder to notice.

Protect yourself with a few habits:

  • Pin versions in your requirements or lock file, and upgrade deliberately, not by accident on a fresh install.
  • Run the spec tests in CI against the pinned versions, and again against the next version when you plan an upgrade. The failures list exactly what moved.
  • Pin the browser-side library too: the CDN URL or bundled JavaScript has its own version, separate from the Python package. A mismatch can make a spec from one version fail in another.
  • Keep the exported spec (JSON) of important reports. If a chart changes after an upgrade, you can diff the old and new specs and see why.

What is real and what is the stand-in

  • Real, read-only: fig.to_dict(), chart.to_dict(), Bokeh's JSON export and the test snippets. They need the libraries installed.
  • Stand-in: the dictionaries in the TryIts mimic a Plotly figure's shape, and the checks are written by hand. A real Plotly figure object has many more fields and defaults, so your assertions on it should target the properties you care about.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why test the figure spec instead of comparing screenshots?

    A spec test asserts facts about the chart (values, labels, scales) and does not break when anti-aliasing or fonts change.
  2. 2. What is the main rule for a bar chart's value axis?

    A truncated bar axis exaggerates differences. Lines and dots can use a non-zero baseline when it is clearly labeled.
  3. 3. What is the danger of a chart with two y-axes?

    Each range is arbitrary, so the visual relationship between the lines is arbitrary too. Stack two charts or index both series to a common base.
  4. 4. A snapshot test fails after a library upgrade. What should you do?

    The diff tells you exactly what moved. Accept it only after you judge the new spec to be right.
  5. 5. Bubble sizes are scaled by radius. A value twice as large will look how many times bigger?

    Area grows with the square of the radius, so doubling the radius quadruples the area. Scale by area to match the value.

Summary

  • Test the spec (dict or JSON) rather than pixels: assert values, labels, lengths, missing data and scale rules.
  • Add a snapshot of the sorted, rounded spec; review its diff like a code change.
  • Bars start at zero; be wary of dual axes; fight overplotting with transparency, jitter or binning.
  • Scale bubbles by area, sort categories, avoid many-slice pies and 3D effects.
  • Pin both Python and browser-side library versions and run spec tests in CI before upgrades.
  • The library calls are read-only here; the Python stand-ins apply the same checks to Plotly-shaped dictionaries.

Next: Project: Sales Dashboard.