Skip to content
Tutorials Center
My learning

Topics

Lesson 19 of 36 ยท Notebooks and Jobs

Notebooks and Magic Commands

Learn how Databricks notebooks work: cells, the %sql, %python, %md, %sh and %run magic commands, dbutils helpers, display(), and sharing state between languages.

A Databricks notebook is the place where most people meet the platform. It looks like a Jupyter notebook: a list of cells, each with code and its output underneath. The difference is what sits behind it. Every notebook attaches to a compute resource (a cluster or serverless compute) that already has Spark running, so the variable spark exists before you type a line.

This lesson shows the cell model, the magic commands that switch language, the dbutils helper object, and the habits that keep notebooks maintainable.

Databricks itself cannot run on this site, so Databricks code below is read-only and described. The runnable blocks are plain Python stand-ins that reproduce the idea (a cell dispatcher, a file-system helper) and are labelled as such.

Cells and the default language

When you create a notebook you pick a default language: Python, SQL, Scala or R. Every cell runs in that language unless it starts with a magic command. Cells share one running session, so a variable defined in cell 1 is visible in cell 5 as long as the cluster stays attached.

Python
# Cell 1 (Python notebook)
orders = spark.read.table("main.sales.orders")
orders.count()

The last expression of a cell is shown as its output. display(df) renders a DataFrame as an interactive table with sorting and built-in charts; df.show() prints plain text.

Python
display(orders.limit(10))

Running a cell does not save it to a file you can open in an editor. Notebooks are stored in the workspace (and can be synced to Git repos, see Asset Bundles and CI/CD). In "source" format a Python notebook is a normal .py file whose first line is # Databricks notebook source and whose cells are separated by # COMMAND ----------.

Magic commands

A magic command is a first line starting with %. It changes how that single cell is interpreted.

Magic What the cell does
%python, %sql, %scala, %r Run the cell in that language
%md Render Markdown (headings, lists, links)
%sh Run shell commands on the driver node only
%run ./other_notebook Run another notebook inline, sharing its definitions
%pip install pkg Install a Python library for this notebook session
%fs ls /Volumes/... Shortcut for dbutils.fs commands

A SQL cell inside a Python notebook:

SQL
%sql
SELECT region, SUM(amount) AS revenue
FROM main.sales.orders
GROUP BY region
ORDER BY revenue DESC

The result of a %sql cell is available to Python as _sqldf (a DataFrame), so you can continue in Python without writing the query again. The other direction is spark.sql("..."), which returns a DataFrame you can keep transforming.

Because each notebook is one session, %pip install should be near the top. Installing a library restarts the Python process, so any variables you created earlier are lost.

Sharing data between languages

Variables do not cross languages. A Python variable is not visible in a Scala cell. What is shared is the Spark catalog: register a temporary view in one language and read it from another.

Python
orders.createOrReplaceTempView("orders_v")
SQL
%sql
SELECT COUNT(*) FROM orders_v

A temporary view lives only for this Spark session. For anything that must outlive the notebook, write a table (see Delta Lake basics) instead.

dbutils: the notebook toolbox

dbutils is a built-in object with utilities for files, secrets, widgets and notebook control.

Python
dbutils.fs.ls("/Volumes/main/sales/raw")            # list files
dbutils.fs.head("/Volumes/main/sales/raw/a.csv")    # first bytes of a file
dbutils.secrets.get(scope="app", key="api-token")   # read a secret
dbutils.notebook.run("child", 600, {"day": "2026-01-01"})
dbutils.notebook.exit("done")                        # return a value to the caller
dbutils.help()                                       # list everything

Secrets are never printed: Databricks replaces the value with [REDACTED] in notebook output. Widgets are covered in Widgets and Parameters.

To make the idea of "a file helper that hides the storage" concrete, here is a tiny stand-in. It is not dbutils; it just mimics ls and head over an in-memory file tree.

Python Stand-in: a dbutils.fs look-alike

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

How a cell is dispatched

Under the hood a notebook looks at the first line of each cell, picks the interpreter, and runs the rest. The stand-in below does the same with three tiny "languages" so you can see why a missing magic line means "use the default".

Python Stand-in: dispatching cells by magic command

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Good notebook habits

  • Keep the top cell for setup: imports, %pip, widgets, constants.
  • Name things: a Markdown cell before each logical step reads like documentation.
  • Do not leave display() of huge tables in production notebooks; add .limit().
  • Move shared code out: use %run for small helpers, but prefer Python files in a repo imported as modules when logic grows.
  • Restart cleanly before a job: "Run all" from a fresh state catches hidden dependencies on cell order.
  • Never paste a token into a cell; use secrets.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A Python notebook has a cell beginning with %sql. What happens?

    A magic command changes the language of that cell only.
  2. 2. Where do %sh commands run?

    %sh executes on the driver, so it does not see files local to workers.
  3. 3. How can a Scala cell read data prepared in a Python cell?

    Variables are per language, but views and tables live in the shared Spark catalog.
  4. 4. What does installing a library with %pip do to existing Python variables?

    Put %pip at the top, before you create variables.

Summary

  • A notebook is a list of cells attached to compute; spark and dbutils are ready to use.
  • Magic commands (%sql, %md, %sh, %run, %pip, %fs) change what one cell does.
  • Languages share the Spark catalog (views, tables), not variables.
  • dbutils covers files, secrets, widgets and notebook-to-notebook calls.

Next: Widgets and Parameters