Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 36 ยท Getting Started

Your First Databricks Notebook

Write your first Databricks notebook: Python, SQL and Markdown cells, magic commands, display(), the built-in spark session and dbutils helpers.

A Databricks notebook is a document made of cells. Each cell holds code or text, and you run cells one at a time. Output (tables, charts, printed text) appears under the cell. Notebooks are the main place where people explore data and prototype pipelines, and they can later be scheduled as jobs.

The notebook is attached to compute, which does the actual work. In Free Edition that is serverless compute, attached for you. In a paid workspace you choose a cluster at the top of the notebook. Variables you define live in that compute session and are lost when the session ends, so re-running cells from the top is a healthy habit.

Cell types and magic commands

Every notebook has a default language (Python, SQL, Scala or R). You can override the language for one cell by starting it with a magic command:

Magic Meaning
%python Run the cell as Python
%sql Run the cell as SQL
%md Render the cell as Markdown text
%sh Run shell commands on the driver (not on serverless)
%run ./other_notebook Run another notebook first and keep its definitions
%pip install pkg Install a Python library for this session

A notebook with a Python default might look like this (read-only Databricks code, shown as one block with cell separators):

Python
# Cell 1 (Python)
greeting = "hello lakehouse"
print(greeting.upper())

# Cell 2 (starts with %sql)
# %sql
# SELECT current_catalog(), current_schema();

# Cell 3 (starts with %md)
# %md
# ## Notes
# This cell becomes formatted text.

Every Python cell in a notebook shares the same variables, so a value created in cell 1 is available in cell 5. SQL cells can read tables but cannot see your Python variables directly. To pass a value, use a table, a temporary view or a widget (widgets are covered later in the course on the Databricks track page).

What is already there

You do not create a Spark session in a notebook. Databricks defines these for you:

  • spark: the SparkSession, entry point for DataFrames and SQL.
  • sc: the SparkContext (not available on serverless compute).
  • dbutils: helper utilities for files, secrets, widgets and notebooks.
  • display(...): renders a DataFrame as an interactive table or chart.
Python
df = spark.range(5)           # a DataFrame with ids 0..4
display(df)                   # interactive table with chart options
print(df.count())             # 5

display() works only inside Databricks. In plain PySpark you would use df.show(), which prints text. In a notebook, prefer display() because it lets you sort the table and switch to a chart without writing more code.

dbutils in two minutes

dbutils groups helpers into modules. Three you will use early:

Python
dbutils.fs.ls("/Volumes/main/default/my_volume")   # list files
dbutils.widgets.text("country", "US")              # create an input box
country = dbutils.widgets.get("country")           # read it
dbutils.help()                                     # list all modules

Never hard-code passwords. dbutils.secrets.get(scope, key) fetches a secret and Databricks redacts it in output.

Stand-in: a tiny notebook engine

To see how "cells share state" works, the stand-in below runs a list of code cells in one shared namespace, exactly like a notebook session. It is plain Python exec, not Databricks.

Python Stand-in: cells that share variables

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

What happens when a session restarts? Variables disappear, and cells that depended on them fail. The stand-in below simulates that and shows why you should be able to "Run all" from top to bottom.

Python Stand-in: state is lost on restart

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Output, display and charts

Printing with print() gives plain text. For tables, display(df) gives a grid with sorting; below it you can add a chart (bar, line, scatter and more) and configure axes in the interface. Charts are a view of the displayed data, so they are only as complete as that data. If you display df.limit(1000), your chart shows at most 1000 rows.

Collaboration features

  • Comments on cells for discussion.
  • Version history so you can restore an older state.
  • Sharing and permissions per notebook or folder.
  • Git folders to sync notebooks with a repository (see Files, volumes and Git folders).

Good notebook habits

  1. Put a Markdown title cell at the top that says what the notebook does.
  2. Keep imports and configuration in the first code cell.
  3. Run from the top after edits to make sure it works from a clean session.
  4. Move reusable code to functions or another notebook or a Python file.
  5. Clear secrets and large outputs before sharing.

Common mistakes

  • Relying on cells that were run out of order; the notebook looks fine but fails when run fresh.
  • Using .collect() or .toPandas() on a huge DataFrame, which pulls everything into the driver's memory.
  • Forgetting that %sh runs on the driver machine, not your laptop.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How do you run one cell as SQL in a Python notebook?

    Magic commands such as %sql override the notebook's default language for a single cell.
  2. 2. Which object is already defined for you in every Databricks notebook?

    The SparkSession is available as spark, so you do not create it yourself.
  3. 3. Why should you be able to 'Run all' from a clean session?

    Notebook state lives in the compute session. If a notebook only works when cells were run in a special order, it is fragile.
  4. 4. What does display(df) do that print(df) does not?

    display() is a Databricks function that shows sortable tables and lets you switch to charts.

Summary

  • A notebook is an ordered list of cells attached to compute; variables live in the session.
  • Magic commands (%sql, %md, %run, %pip) change what a cell does.
  • spark, dbutils and display() are ready to use without setup.
  • Write notebooks so that "Run all" works from a fresh session.

Next: A Tour of the Workspace