Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 39 ยท Reproducibility

Hidden State and Restart and Run All

Understand why notebooks silently depend on cell execution order, how hidden state breaks reproducibility, and how Restart and Run All plus a few habits fix it.

The most common reason a notebook cannot be reproduced has nothing to do with libraries. It is hidden state: variables that exist in the kernel's memory because of cells you ran earlier, edited, deleted or ran in a different order. The file on disk shows one story; the kernel remembers another.

This lesson explains how the problem arises, how to detect it, and the habits that prevent it. Because Jupyter cannot run here, the runnable cells simulate a kernel in plain Python so you can watch hidden state appear and disappear.

How a kernel remembers

A kernel holds one namespace, a dictionary of names to values. Each time you run a cell, its code executes against that same dictionary. Nothing resets it between cells. The number in the margin, In [7], is the order of execution, not the position in the file.

flowchart TD
  A["Run cell A: x = 10"] --> N["Namespace: x=10"]
  N --> B["Run cell B: y = x * 2"]
  B --> N2["Namespace: x=10, y=20"]
  N2 --> C["Delete cell A, keep running"]
  C --> N3["x and y still exist in memory"]

Deleting or editing a cell does not undo what it already did to the namespace.

Four ways hidden state sneaks in

  1. Out-of-order execution. You run cell 5 before cell 3, because cell 3 was edited later. Cell 5 uses an old value of a variable.
  2. Deleted cells. The cell that defined threshold is gone, but the variable is still alive, so everything that uses it still works, until someone else runs the notebook.
  3. Mutated objects. A cell appends to a list or modifies a DataFrame in place. Running it twice gives different results.
  4. Stale imports. You changed a module on disk, but import mymodule is cached for the kernel's lifetime.

Simulate it

The stand-in below is a tiny kernel: a shared dictionary and a function that runs "cells" in a chosen order. You can see how the final answer depends on the order, and what a fresh run gives.

Python A toy kernel with one shared namespace

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The session and the fresh run disagree: 54.0 against 30.0. The notebook file only contains discount = 0.5, so a reader who opens it gets 30.0, and your 54.0 can never be reproduced from the file.

The cure: Restart and Run All

Every notebook interface has a command that clears the namespace and runs every cell top to bottom, in file order. In classic Notebook and JupyterLab it is Kernel, Restart Kernel and Run All Cells. If that run succeeds and the results are what you expect, the notebook is self-contained.

Make it a habit:

  • Do it before you share or commit a notebook.
  • Do it whenever a result surprises you. Half of the "bugs" disappear.
  • Do it automatically in CI, with jupyter nbconvert --to notebook --execute notebook.ipynb, so a broken notebook fails the build (see exporting with nbconvert).

A restart also clears memory, which is useful after loading huge data.

Habits that prevent hidden state

  • Define before use, top to bottom. Imports and constants in the first cells, then loading, then processing.
  • Prefer functions to loose variables. A function sees only its arguments and returns a value, so it cannot rely on stray globals.
  • Avoid in-place mutation across cells. Build a new object (df2 = df.assign(...)) rather than changing df in three different cells.
  • Never leave a cell you rely on out of the file. If a value matters, it must be defined in a cell that exists.
  • Reload modules deliberately. For your own modules, use the autoreload extension (see the next section).
Python Why running a mutating cell twice is dangerous

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The unsafe cell keeps growing the list; the safe cell gives the same answer however often you run it. A cell that gives the same output on every run is called idempotent, and it is one of the best properties a notebook cell can have.

Reloading your own modules

If you moved code into a .py file (see moving code into modules), Python caches the import. Enable autoreload in the first cell to pick up edits:

Python
%load_ext autoreload
%autoreload 2

Mode 2 reloads every module before each execution. It works well for simple functions but can confuse class instances created before the reload, so restart the kernel when something looks inconsistent.

Spotting hidden state quickly

Execution counts reveal a lot. If the numbers in the margin are not 1, 2, 3, 4 from top to bottom, cells ran out of order. A notebook with gaps or repeats should be re-run before it is trusted. Some teams add a pre-commit check that verifies the counts are sequential or that outputs are cleared.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is hidden state in a notebook?

    Variables set by deleted, edited or out-of-order cells still live in the namespace, so results may depend on things the file does not show.
  2. 2. What does Restart Kernel and Run All Cells do?

    It starts from an empty namespace and executes the file in order, exactly what a new reader will see.
  3. 3. Which cell is idempotent?

    Assigning a fresh value gives the same state every time; the others change when run again.
  4. 4. What does the number in In [7] mean?

    It is the execution counter, which can differ from the visual order.

Summary

  • A kernel has one shared namespace that outlives edits and deletions of cells.
  • Hidden state comes from out-of-order runs, deleted cells, in-place mutation and cached imports.
  • Restart and Run All proves a notebook is self-contained; do it before sharing.
  • Write idempotent cells, define things before using them, and prefer functions.

Next: Seeds and Deterministic Results