Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 36 ยท Platform and Workspace

Files, Volumes and Git Folders

Learn where files live in Databricks: Unity Catalog volumes, workspace files, legacy DBFS and cloud storage paths, and how Git folders keep your code in version control.

Data is not only tables. You will handle CSV uploads, JSON logs, images, model files and Python scripts. Databricks has several places to put files, and picking the right one avoids permission surprises later. This lesson maps them and then shows how to keep your code safe in Git.

The places files can live

Location Path style Use for
Unity Catalog volume /Volumes/catalog/schema/volume/... Data files governed like tables (recommended)
Workspace files /Workspace/Users/you/... Code, small config and helper files
Cloud object storage s3://, abfss://, gs:// Direct access to your bucket or container
DBFS dbfs:/... Legacy; avoid for new work
Driver local disk /tmp/... Temporary scratch, lost when compute stops

Volumes are the modern answer. A volume is a catalog object (so it has the same catalog.schema naming and permissions as a table) that points to storage and holds any kind of file. Managed volumes are stored by Databricks; external volumes point to a location you own.

SQL
CREATE VOLUME IF NOT EXISTS main.default.landing;

Then in Python (read-only Databricks code):

Python
path = "/Volumes/main/default/landing/orders.csv"

df = (spark.read
      .option("header", True)
      .option("inferSchema", True)
      .csv(path))
display(df)

dbutils.fs.ls("/Volumes/main/default/landing")

Notice that the path looks like a normal file path, so plain Python libraries (open, pandas.read_csv) can read it too on most compute. Permissions come from READ VOLUME and WRITE VOLUME grants.

Uploading a file

In the interface you can open a volume in the Catalog Explorer and upload a file, or use "Create or modify table" from an upload to turn a CSV directly into a Delta table. For automation, use dbutils.fs.cp or the Databricks CLI and SDK.

DBFS, the old file system

DBFS (Databricks File System) is an older abstraction layer over storage, with paths like dbfs:/FileStore/.... You will meet it in old tutorials. Public root access to it is restricted in modern workspaces, and Unity Catalog governance does not apply to it as it does to volumes. Use volumes for new projects.

Reading directly from cloud storage

If your data sits in a bucket, you can register an external location (a credential plus a storage path, governed by Unity Catalog) and then read it with a URL:

Python
df = spark.read.format("parquet").load("s3://my-company-lake/raw/events/")

Access goes through governed credentials; you do not paste cloud keys into notebooks.

Git folders

Notebooks stored only in the workspace are hard to review and version. Git folders (formerly Repos) clone a Git repository into the workspace so you can:

  • commit, push and pull from the notebook interface,
  • work on branches and open pull requests on GitHub, GitLab, Azure DevOps and others,
  • run the same code in dev and in scheduled jobs by pointing the job at a branch or tag,
  • keep notebooks and plain .py files together.

Notebooks are saved in Git as source files (for example a .py file with special comment markers, or .ipynb), so diffs are readable.

A typical flow:

flowchart LR
    A["Create Git folder<br/>from repo URL"] --> B["Create a branch"]
    B --> C["Edit and run notebooks"]
    C --> D["Commit and push"]
    D --> E["Pull request, review, merge"]
    E --> F["Job runs main branch"]

The Git provider is the source of truth; the workspace folder is a working copy.

Do not edit the same notebook in the production Git folder by hand. Make changes on a branch and merge them, so production code is always something that was reviewed.

Stand-in: a tiny volume with path rules

The stand-in below creates an in-memory "volume" and enforces the pattern /Volumes/catalog/schema/volume/..., including a read-only permission. It is only a model of the naming and permission idea.

Python Stand-in: paths and permissions of a volume

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second stand-in reads a small CSV and turns it into a table, which is what "create table from upload" does: parse text, infer simple types, store rows.

Python Stand-in: CSV text to typed rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Choosing a location

  • Data other people query: volume or a Delta table.
  • Python helper modules and configuration: workspace files inside a Git folder.
  • Huge existing lake: external location over the bucket.
  • Temporary intermediate files: driver /tmp, remembering they vanish.

Common mistakes

  • Writing important data to driver local disk and losing it when compute stops.
  • Using DBFS paths from old tutorials in a Unity Catalog workspace.
  • Committing secrets or large data files to a Git folder.
  • Editing production notebooks directly instead of using branches.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which location is recommended for governed data files in a Unity Catalog workspace?

    Volumes are catalog objects with permissions, so files are governed like tables.
  2. 2. What does a volume path look like?

    Volumes are addressed as /Volumes/<catalog>/<schema>/<volume>/...
  3. 3. What is the main benefit of Git folders?

    Git folders connect workspace code to version control so changes are reviewed and reproducible.
  4. 4. What happens to files on the driver's local disk when compute stops?

    Local disk is temporary. Persist anything important to a volume or table.

Summary

  • Volumes hold governed files at /Volumes/catalog/schema/volume/....
  • Workspace files and Git folders are for code; DBFS is legacy; driver disk is temporary.
  • External locations expose existing cloud storage with governed credentials.
  • Use branches and pull requests so production code is reviewed.

Next: Spark Architecture