Skip to content
Tutorials Center
My learning

Topics

Lesson 7 of 39 ยท Notebooks and Kernels

Execution Order and Hidden State

Learn why Jupyter notebooks can give different results depending on the order cells ran, how hidden state arises, and the habits that keep notebooks reliable.

A notebook looks like a document you read from top to bottom, but it runs in whatever order you click. The kernel does not care about the layout on the page: it simply executes the cell you tell it to, using whatever variables currently exist. That freedom is what makes notebooks pleasant for exploration, and it is also the number one reason notebooks break for other people. A result you see on your screen may depend on a variable you deleted an hour ago.

This lesson names the problem (hidden state), shows it happening, and gives you reliable habits to avoid it.

We cannot run a real kernel here, so the examples simulate cells as strings run in a shared namespace, exactly as the kernel does. The behaviour you see is the behaviour of a real notebook.

Out-of-order execution

Imagine a notebook with three cells:

Position Code
1 count = 10
2 count = count + 5
3 print(count)

Run top to bottom and you get 15. Now suppose you run cell 2 twice by accident (it is easy: press Ctrl+Enter twice). count becomes 20, and cell 3 prints 20 although the page still says "10, plus 5". The page shows a story that is no longer true.

Python Stand-in: running the same cell twice

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The output you get depends on the history of what was run, not on the text of the notebook. That is the definition of hidden state.

Four classic ways hidden state appears

  1. Re-running a cell that changes a variable in place (like count += 5, data.append(x), or df = df.dropna()). The second run starts from the changed value.
  2. Deleting or editing a cell after it has run. The variable it created still exists in memory, so later cells keep working. A new reader who runs the file from the start gets NameError.
  3. Running cells in a different order than they appear: a cell near the bottom defines something that a cell near the top then uses.
  4. Renaming a variable but forgetting one use. The old name remains in kernel memory, so no error shows up until restart.

The following stand-in demonstrates the second one. We "delete" the cell that defined rate, but the kernel still remembers the value, so the result looks fine until a restart reveals the truth:

Python Stand-in: a deleted cell leaves a ghost variable

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Detecting the problem

The prompt numbers [1], [2], ... are your first clue. If they are not in ascending order down the page, some cells ran out of order. A cell with an old number between newer ones is a warning sign.

A more reliable test is mechanical: Kernel โ†’ Restart Kernel and Run All Cells. If every cell succeeds on a fresh kernel, and you get the same results, the notebook is sound. If you see an error, you found hidden state.

Stand-in "lint": check that execution counts increase from top to bottom, the same simple test you can do by eye:

Python Stand-in: check that execution counts are in order

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Habits that keep notebooks honest

  • Restart and run all regularly, and always before sharing or committing.
  • Prefer cells that create new names rather than mutate old ones: write clean = raw.dropna() instead of re-assigning raw repeatedly. Then a cell can be run twice with the same result. This property is called idempotent.
  • Keep setup at the top: imports, constants and configuration go in the first cell, so a restart rebuilds everything in one step.
  • One idea per cell, with a Markdown note before it. Big cells that do five things are hard to re-run safely.
  • Wrap repeated logic in functions. A function takes inputs and returns outputs, so it has no secret dependence on earlier cells. Define it once, test it, call it often.
  • Do not depend on a cell you will later delete. If you remove a cell, restart and run all straight away.
  • Avoid long-running state in the kernel, like an opened file you never close. Use with blocks so resources close inside the cell.

Here is the "idempotent" idea shown with a stand-in: the mutating cell gives different answers on repeated runs; the non-mutating cell does not.

Python Stand-in: idempotent versus mutating cells

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A note on "reactive" notebooks

Some tools aim to fix this at the root. Reactive notebooks (for example Marimo, and Observable in JavaScript) track which cell uses which variable, and re-run dependent cells automatically, so the page and the state cannot disagree. Classic Jupyter does not do that: it trusts you. Extensions and the practices above fill the gap. See also The .ipynb file format, which stores the execution counts you just used for diagnosing order problems.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is hidden state in a notebook?

    Variables created by earlier runs, deleted cells or out-of-order execution can silently change results.
  2. 2. You delete the cell that defined rate. Later cells still work. Why?

    The kernel keeps the variable. A fresh kernel would raise NameError.
  3. 3. What is the best single test that a notebook is sound?

    A clean top-to-bottom run on a fresh kernel exposes hidden state.
  4. 4. Which cell is idempotent?

    sorted returns a new list and leaves items untouched, so repeated runs give the same value.

Summary

  • The kernel runs whichever cell you choose; page order and execution order can differ.
  • Re-running mutating cells, deleting cells and running out of order create hidden state.
  • Prompt numbers that are not increasing downwards signal out-of-order execution.
  • Restart and Run All is the definitive check; do it before sharing.
  • Make cells idempotent, put setup first, use functions, and keep one idea per cell.
  • Reactive notebooks exist, but classic Jupyter relies on your discipline.

Next: The .ipynb file format