Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 36 ยท Streaming and Production

Asset Bundles, Testing and CI/CD

Ship Databricks work like software: Declarative Automation (Asset) Bundles in YAML, targets for dev and prod, the CLI, unit tests for transforms, and a CI/CD flow.

Clicking a job together in the UI works for a first experiment. It does not work when five engineers change the same pipeline, when you need a test environment, or when someone asks "what exactly is running in production?" The answer is the one you already use for other software: put everything in Git, test it automatically, and deploy through a pipeline.

On Databricks the standard tool is Databricks Asset Bundles (recent releases also call the feature Declarative Automation Bundles; the CLI commands are databricks bundle ...). A bundle describes jobs, pipelines, notebooks, models and permissions as YAML files next to your code.

Databricks cannot run on this site. Bundle files and CLI commands are read-only; runnable stand-ins show how to unit-test transformation logic and how the environment overrides work.

A bundle project

TEXT
orders-etl/
  databricks.yml
  resources/
    orders_job.yml
  src/
    transforms.py
    notebooks/load_orders.py
  tests/
    test_transforms.py

databricks.yml is the entry point:

YAML
bundle:
  name: orders-etl

include:
  - resources/*.yml

variables:
  catalog:
    description: Target catalog
    default: dev

targets:
  dev:
    mode: development          # prefixes resource names with your user, pauses schedules
    default: true
    workspace:
      host: https://dev-workspace.cloud.databricks.com
  prod:
    mode: production
    workspace:
      host: https://prod-workspace.cloud.databricks.com
      root_path: /Workspace/Shared/.bundle/${bundle.name}/${bundle.target}
    variables:
      catalog: prod
    run_as:
      service_principal_name: "orders-etl-sp"

A resource file defines the job:

YAML
resources:
  jobs:
    orders_job:
      name: orders-etl-${bundle.target}
      schedule:
        quartz_cron_expression: "0 30 2 * * ?"
        timezone_id: UTC
      tasks:
        - task_key: load_orders
          notebook_task:
            notebook_path: ../src/notebooks/load_orders.py
            base_parameters:
              catalog: ${var.catalog}
          environment_key: default
      environments:
        - environment_key: default
          spec:
            environment_version: "3"
            dependencies: ["-e ../"]

Variables like ${var.catalog} and ${bundle.target} let one definition serve all environments. The mode: development target adds your name to resource names and pauses schedules, so developers cannot disturb production.

The CLI workflow

Terminal
databricks bundle validate                # check syntax and references
databricks bundle deploy -t dev           # create/update jobs and upload files
databricks bundle run orders_job -t dev   # start a run
databricks bundle deploy -t prod          # usually done by CI, not by hand
databricks bundle destroy -t dev          # remove everything this bundle created

deploy is idempotent: it compares your YAML with what exists and changes only the difference, so the Git repository becomes the source of truth.

Python Stand-in: dev and prod targets overriding defaults

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Make logic testable

The core habit: keep transformations in plain Python functions in src/, not buried in notebook cells. Then tests run on a laptop or CI runner without a cluster.

Python
# src/transforms.py
from pyspark.sql import DataFrame, functions as F

def add_net_amount(df: DataFrame, tax_rate: float) -> DataFrame:
    return df.withColumn("net_amount", F.round(F.col("amount") / (1 + tax_rate), 2))
Python
# tests/test_transforms.py   (run with pytest; Spark via a local session or Databricks Connect)
def test_add_net_amount(spark):
    df = spark.createDataFrame([(1, 120.0)], ["id", "amount"])
    out = add_net_amount(df, 0.2).collect()[0]
    assert out["net_amount"] == 100.0

Pure-Python logic (parsing, validation, business rules) is the easiest to test, as below.

Python Stand-in: unit tests for a transformation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three levels of testing are common: unit tests on functions (fast, in CI), integration tests that deploy to a dev target and run a job on small data, and data quality tests (expectations, row counts) that run in production.

A CI/CD flow

flowchart LR
    A["Pull request"] --> B["Lint + unit tests"]
    B --> C["bundle validate"]
    C --> D["Deploy to dev/test and run integration job"]
    D --> E["Merge to main"]
    E --> F["Deploy to prod as service principal"]

Every change is checked before it can reach production.

A GitHub Actions sketch (read-only):

YAML
name: deploy
on:
  push: { branches: [main] }
jobs:
  deploy-prod:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: databricks/setup-cli@main
      - run: databricks bundle deploy -t prod
        env:
          DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
          DATABRICKS_CLIENT_ID: ${{ secrets.DATABRICKS_CLIENT_ID }}
          DATABRICKS_CLIENT_SECRET: ${{ secrets.DATABRICKS_CLIENT_SECRET }}

The CI system authenticates as a service principal with OAuth client credentials held in the CI secret store. Nobody deploys to production from a laptop. See CI/CD with GitHub Actions for the general idea of pipelines.

Environments and cost

  • Use separate catalogs or workspaces for dev, test and prod; promote code, not data.
  • Give developers small sample data and serverless or tiny clusters; use cluster policies to cap size.
  • Tag jobs (team, project) so the billing system tables can break cost down.
  • Schedule heavy jobs off-peak, prefer job compute or serverless, and use availableNow instead of always-on streams when freshness allows.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the main benefit of Asset Bundles?

    Infrastructure as code: Git is the source of truth.
  2. 2. What does a development-mode target do?

    It isolates developers from production and from each other.
  3. 3. Where should business logic live to be easy to test?

    Functions in modules can be unit tested in CI without a cluster.
  4. 4. Who should the production deployment run as?

    Machine identities keep deployments auditable and independent of people.

Summary

  • Bundles describe jobs, pipelines and settings in YAML with targets for each environment.
  • databricks bundle validate / deploy / run is the workflow; Git is the source of truth.
  • Keep logic in testable modules; use unit, integration and data-quality tests.
  • Deploy from CI as a service principal, and control cost with tags, policies and serverless.

Next: Project: A Medallion Pipeline