Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 39 ยท Cells and Magics

Shell Commands and the System

Run terminal commands inside a notebook with ! and %%bash, capture their output into Python lists, pass variables, install packages safely with %pip, and avoid common shell pitfalls.

Data work involves more than Python: you list files, unzip an archive, check disk space, install a package, call git. Jupyter lets you do all of this without leaving the notebook. In an IPython kernel, a line that starts with an exclamation mark runs in the operating system shell, and the output appears just like any other cell output.

Shell access needs a real kernel on a real machine, so these commands are shown read-only. The stand-ins model the behaviours that surprise people: how output is captured into a list, how variables are substituted, and why !cd does not stick.

The ! prefix

Python
!ls                    # list files (on Windows use !dir)
!pwd                   # print the working directory
!python --version
!head -n 3 data.csv

Each !command starts a new shell process, runs the command, and prints its standard output. The result is a one-way trip: the shell exits afterwards, so nothing that the command changes inside the shell (the current directory, shell variables) survives to the next line.

Capturing the output

Assign the command to a variable to get its output as a Python object instead of printing it:

Python
files = !ls *.csv
files

files is an IPython SList: a list of strings (one per line) with helper properties:

Attribute / method Meaning
files.n or files.nlstr All lines joined with newlines
files.s All lines joined with spaces (handy to pass to another command)
files.grep("2024") Only lines containing the text
files.fields(0, 2) Select whitespace-separated columns
files.sort(1, nums=True) Sort by a column, numerically

Because it is still a list, ordinary Python works: len(files), files[0], loops, comprehensions. This is a powerful bridge: shell tools gather the raw material, Python processes it.

The stand-in imitates an SList so you can try its helpers on sample text:

Python Stand-in: an SList-like result of !command

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Passing Python variables to the shell

You can embed Python values into a shell command in two ways:

Python
name = "sales_2024.csv"
folder = "data"

!echo {name}            # curly braces: any Python expression is inserted
!ls -l {folder}/{name}
!echo $name             # $name: a plain variable name
!echo {len(name)}       # expressions only work in braces

Note: if you need a literal dollar sign for the shell (for example awk '{print $2}'), double it: $$. That is also the reason awk and similar commands are often easier in %%bash cells, where there is no substitution.

Here is a stand-in for the substitution step. Python's string.Template-like logic does the same job:

Python Stand-in: substitute Python values into a command

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last line shows a doubled dollar sign surviving as a single $ for the shell. (Real IPython also treats curly braces specially, so for awk '{print $2}' a %%bash cell is the easier choice.)

Multi-line scripts: %%bash and %%script

When you need several commands, loops or pipes, use a cell magic:

Terminal
%%bash
cd data
for f in *.csv; do
  echo "$f: $(wc -l < "$f") lines"
done

%%bash runs the whole cell as one Bash script (so cd works inside that script), shows the output, and does not substitute {} or $ the way ! lines do. Similar magics exist for other interpreters: %%sh, %%script python3, %%script perl, %%ruby. Add --bg to run the script in the background so the notebook stays responsive:

Terminal
%%bash --bg --out result
sleep 20 && echo done

--out stores the output in a Python variable. Use --err for standard error.

Directory changes: %cd versus !cd

!cd data does nothing useful, because the shell that changed directory exits immediately. To change the kernel's working directory use the line magic %cd, or Python's os.chdir:

Python
%cd data
%pwd

The stand-in shows why the subprocess approach loses the change:

Python Stand-in: why !cd does not stick

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Installing packages: %pip and %conda

A classic trap: !pip install pandas may install into a different Python than the kernel uses (see Kernels explained). Use the magics, which install into the kernel's own environment:

Python
%pip install pandas
%conda install -y numpy       # only inside a conda environment

After installing a package for the first time, restart the kernel if you had already imported an older version, because Python caches imported modules. Better still, install your dependencies from a file (%pip install -r requirements.txt) and record them for the project; reproducibility is covered in the next chapters.

Safety notes

  • Shell commands run with your permissions. Never run ! cells from a notebook you do not trust.
  • Avoid destructive commands (rm -r, del) with variables you have not checked; print the command first.
  • Commands differ between systems: ls/dir, cat/type. If a notebook must work everywhere, prefer Python's pathlib, shutil and subprocess.
  • Shell output goes into the notebook: large listings bloat the file.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does files = !ls *.csv store in files?

    Assigning a ! command captures its output as an IPython SList, which is a list of strings.
  2. 2. Why does !cd data not change the directory of later cells?

    Every ! line runs in a new subprocess. Use %cd or os.chdir to change the kernel's working directory.
  3. 3. Which is safer for installing a package into the kernel's environment?

    %pip targets the Python that runs the kernel, avoiding mismatched environments.
  4. 4. How do you use a Python variable name in a ! command?

    IPython substitutes and $variable in ! lines before running them.

Summary

  • !command runs one shell command; files = !command captures output as an SList of lines.
  • Insert Python values with or $name; double $$ for a literal dollar.
  • %%bash and %%script run multi-line scripts; --bg, --out and --err give background runs and captured output.
  • Use %cd (not !cd) to change directory, and %pip or %conda to install into the right environment.
  • Restart the kernel after installing a package you had already imported.
  • Treat shell cells as code that runs with your permissions, and prefer portable Python for cross-platform notebooks.

Next: Plots in notebooks