Skip to content
Tutorials Center
My learning

Topics

Lesson 25 of 36 ยท Unity Catalog and Governance

Volumes, Storage Credentials and External Locations

Govern files in Databricks: managed and external volumes, /Volumes paths, storage credentials, external locations, and how path-based access is checked.

Tables govern structured data. But data teams also handle CSV drops, JSON logs, images, model files and ZIP archives. Unity Catalog governs those with volumes, and it governs access to cloud storage itself with storage credentials and external locations.

Databricks cannot run on this site. SQL and paths below are read-only; the runnable stand-ins model path checking and file listing in Python.

Volumes

A volume is a securable object inside a schema that represents a directory of files in cloud storage. You address files by a POSIX-style path:

TEXT
/Volumes/<catalog>/<schema>/<volume>/<path>/<file>
/Volumes/prod/sales/raw/2026/01/orders.csv

Create and use one:

SQL
CREATE VOLUME prod.sales.raw COMMENT 'Landing zone for partner files';
GRANT READ VOLUME  ON VOLUME prod.sales.raw TO `analysts`;
GRANT WRITE VOLUME ON VOLUME prod.sales.raw TO `ingest-service`;
Python
df = spark.read.option("header", True).csv("/Volumes/prod/sales/raw/2026/01/")
dbutils.fs.ls("/Volumes/prod/sales/raw")

Because the path works like a normal file path, open(), pandas.read_csv(), os.listdir() and shell commands also work on a volume from a notebook (they read through the same governed mount).

Managed volume External volume
Storage Unity Catalog-managed location A path in an external location you registered
Drop behaviour Files are deleted (after retention) Files remain
Typical use New landing zones, scratch and ML artifacts Existing buckets also used by other tools

Storage credentials and external locations

To reach your own cloud storage, Unity Catalog uses two objects:

  • A storage credential wraps a cloud identity (an IAM role on AWS, a managed identity or service principal on Azure, a service account on Google Cloud). Only administrators create these.
  • An external location pairs a storage URL with one credential: "this path is reachable using that identity."
SQL
CREATE EXTERNAL LOCATION sales_lake
  URL 's3://company-lake/sales'
  WITH (STORAGE CREDENTIAL aws_sales_role);

GRANT READ FILES, CREATE EXTERNAL TABLE ON EXTERNAL LOCATION sales_lake TO `data-engineers`;

Users never see cloud keys. They receive privileges on the external location object, and Databricks vends short-lived tokens behind the scenes. That is how one IAM role is shared safely among many people.

Key rules:

  • External locations may not overlap in ways that make path ownership ambiguous; a path is covered by the most specific location.
  • Tables and volumes created LOCATION '...' need the user to hold CREATE EXTERNAL TABLE or CREATE EXTERNAL VOLUME on the covering location.
  • Avoid granting READ FILES on a broad location to everyone: it bypasses table-level governance for files in that path.
flowchart LR
    U["User / job"] -->|"privilege on"| EL["External location"]
    EL -->|"uses"| SC["Storage credential"]
    SC -->|"assumes"| ID["Cloud identity (role / managed identity)"]
    ID --> ST[("Cloud storage path")]

Users hold privileges on the external location; the credential is hidden.

Checking a path

When code touches a path, Databricks finds which governed object covers it and checks privileges. The stand-in below models the longest-prefix rule and the check.

Python Stand-in: which location covers this path?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Working with volume files in Python

Because volumes look like folders, ordinary Python file code applies. Here is a stand-in that uses a real temporary directory in the runner (Pyodide has an in-memory file system) to mimic a landing-zone ingest: read each file, skip ones already processed.

Python Stand-in: ingesting new files from a landing folder

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

(The real tool for this pattern is Auto Loader, which tracks processed files for you.)

Practical advice

  • Prefer volumes for files and tables for structured data; avoid raw s3:// or abfss:// paths in notebooks, because paths bypass table lineage and make permissions harder to see.
  • Use managed volumes unless another system must own the files.
  • Name volumes by purpose: raw, landing, exports, artifacts.
  • Give ingestion jobs WRITE VOLUME only on the landing volume and give analysts READ VOLUME at most.
  • The legacy DBFS root (dbfs:/) and /mnt mounts are not governed per user; migrate to volumes.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a path /Volumes/prod/sales/raw/a.csv tell you?

    The path is /Volumes/catalog/schema/volume/file.
  2. 2. What is an external location?

    It maps a storage path to the credential used to access it.
  3. 3. You drop an external volume. What happens to its files?

    Only the metadata is removed for external objects.
  4. 4. Why avoid raw s3:// paths in notebooks?

    Governed objects give you permissions, lineage and audit.

Summary

  • Volumes govern files; paths look like /Volumes/catalog/schema/volume/....
  • Storage credentials hold cloud identities; external locations bind them to paths; users get privileges on locations.
  • Managed volumes are cleaned up on drop; external volumes leave files in place.
  • Prefer governed paths to raw cloud URLs and to legacy DBFS mounts.

Next: Lineage, Row Filters and Column Masks