Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 39 ยท Collaboration and Deployment

Parameterized Notebook Runs with Papermill

Turn one notebook into a reusable report: tag a parameters cell, run it with Papermill for different dates or regions, collect outputs, and schedule the runs.

You wrote a notebook that analyzes sales for March. Next month you want April, and next quarter every region separately. Copying the notebook and editing a variable in each copy leads to dozens of slightly different files. Papermill solves this: it runs a notebook as a function. You mark one cell as the place where parameters live, then execute the notebook many times with different values and save each executed copy.

Papermill is not installable on this site, so the command-line and Python usage below is read-only. The runnable cells implement the key mechanism, injecting a parameters cell into a notebook structure, so you can see how it works.

Step 1: tag a parameters cell

Create a code cell near the top with default values and give it the tag parameters (JupyterLab's Property Inspector, right sidebar, has a tag editor):

Python
# tag this cell: parameters
region = "north"
month = "2024-03"
min_order = 0

The values here are defaults. They also make the notebook run on its own when you open it normally. Keep parameters simple: strings, numbers, booleans, lists and dictionaries, which can be written as plain literals.

The rest of the notebook uses these variables, for example to filter a data frame.

Step 2: run it from the command line

Terminal
pip install papermill
papermill report.ipynb out/report-south.ipynb -p region south -p month 2024-04
papermill report.ipynb out/report.ipynb -f params.yaml
papermill report.ipynb out/report.ipynb -y "region: east
month: 2024-05"

Papermill reads report.ipynb, inserts a new cell right after the tagged one containing your values, runs everything in a fresh kernel, and writes the executed notebook to the output path. The original is never changed. The output is an ordinary notebook with results, so you can open it, convert it with nbconvert, or archive it as the record of that run.

From Python, to loop over many runs:

Python
import papermill as pm

for region in ["north", "south", "east", "west"]:
    pm.execute_notebook(
        "report.ipynb",
        f"out/report-{region}.ipynb",
        parameters={"region": region, "month": "2024-04"},
    )

What Papermill injects

The inserted cell is tagged injected-parameters and contains assignments:

Python
# Parameters
region = "south"
month = "2024-04"

Because it comes after the defaults, it overrides them, and because it is a visible cell, the executed notebook shows exactly which values produced it. That is excellent documentation.

flowchart LR
  T["report.ipynb with tagged parameters cell"] --> P["papermill -p region south"]
  P --> I["Insert 'injected-parameters' cell"]
  I --> K["Execute in a fresh kernel"]
  K --> O["out/report-south.ipynb"]

The input notebook stays unchanged; each run produces its own executed copy.

Build the injection yourself

Python Inject a parameters cell (stand-in for Papermill)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second and third runs print different text from the same notebook. A missing parameters tag raises an error here; real Papermill instead inserts the parameters at the top of the notebook, so a forgotten tag is easy to miss, and the defaults in your notebook then silently win. Always check that the injected cell appears in the output.

Type conversion and the command line

On the command line every -p value is a string unless Papermill can parse it. Papermill tries to convert 10 to an integer and true to a boolean, but if you need a string that looks like a number, use -r (raw) or write it in a YAML file with quotes. In a parameter file:

YAML
region: south
month: "2024-04"      # quoted so it stays a string
min_order: 50
regions: [north, south]

Reading results from a run

You often want a number back from a run, not just a notebook. Papermill's companion library scrapbook records named values from inside the notebook (sb.glue("total", total)), and you read them from the executed copy later. A simpler way is to have the notebook write a file (CSV, JSON) to an output folder that includes the parameters in its name.

Python Collect results from many runs

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In a real setup each call would be pm.execute_notebook(...), and the numbers would come from files or scrapbook values.

Scheduling and failure handling

Papermill pairs well with a scheduler: cron, GitHub Actions, Airflow (it has a Papermill operator), Prefect or a plain loop. Remember the practical points:

  • Every run is isolated, so parameter combinations can run in parallel on separate machines.
  • On an error the executed copy is still written, with a traceback in the failing cell. Open it to debug. Pass --log-output to stream progress into logs.
  • Use a pinned environment (see pinning dependencies) so runs on different days stay comparable.
  • Use --cwd to set the working directory when the notebook reads relative paths.
  • Save executed notebooks with the date or parameters in the file name to keep an audit trail.

When not to use Papermill

If the notebook only needs one or two inputs interactively, widgets or a dashboard (see Voila) may be a better fit. If the logic is complex, move it into a function in a module and call that from a script; notebooks should then be thin presentation layers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you tell Papermill where parameters go?

    The cell tagged parameters holds defaults; Papermill inserts the new values right after it.
  2. 2. Does Papermill modify the input notebook?

    The input stays unchanged; every run produces its own executed copy.
  3. 3. What is the name of the tag on the cell Papermill inserts?

    That cell shows exactly which values produced the executed notebook.
  4. 4. What happens if a cell raises an error during a Papermill run?

    The partial executed notebook is kept so you can inspect the error.

Summary

  • Tag one cell parameters to hold defaults; run the notebook with papermill in.ipynb out.ipynb -p name value.
  • Papermill inserts an injected-parameters cell, executes in a fresh kernel and writes a separate output notebook.
  • Quote string values that look like numbers; check that the injected cell appears.
  • Collect results through files or scrapbook, and schedule runs with any scheduler.

Next: Sharing with JupyterHub and Binder