Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 36 ยท Streaming and Production

Declarative Pipelines (Delta Live Tables)

Define data pipelines declaratively in Databricks: streaming tables, materialized views, expectations for data quality, AUTO CDC, and a runnable dependency stand-in.

Writing a pipeline by hand means deciding the order of steps, managing checkpoints, retries, schema changes and quality checks. Declarative pipelines (the product formerly called Delta Live Tables, DLT, now part of Lakeflow) flip this: you declare what each table should contain, and Databricks figures out the execution order, incremental processing, retries and monitoring.

Declarative pipelines cannot run on this site. The code is read-only; runnable blocks are plain Python stand-ins that reproduce dependency resolution and expectations.

Two kinds of dataset

  • Streaming table: processes only new input rows incrementally (append-oriented). Use it for ingestion and for transformations of event streams.
  • Materialized view: a query result that Databricks keeps up to date, incrementally when it can. Use it for aggregations, joins and gold tables.

(Older material uses LIVE TABLE; you will see both. Prefer the newer names.)

SQL version

SQL
CREATE OR REFRESH STREAMING TABLE bronze_orders
AS SELECT *, _metadata.file_path AS source_file
FROM STREAM read_files('/Volumes/prod/sales/raw/orders/', format => 'json');

CREATE OR REFRESH STREAMING TABLE silver_orders (
  CONSTRAINT valid_id    EXPECT (order_id IS NOT NULL) ON VIOLATION DROP ROW,
  CONSTRAINT positive    EXPECT (amount > 0)
)
AS SELECT order_id, customer_id, CAST(amount AS DECIMAL(12,2)) AS amount, order_ts
FROM STREAM bronze_orders;

CREATE OR REFRESH MATERIALIZED VIEW gold_daily_revenue
AS SELECT date(order_ts) AS day, SUM(amount) AS revenue
FROM silver_orders GROUP BY date(order_ts);

Python version

Python
from pyspark import pipelines as dp
from pyspark.sql import functions as F

@dp.table
def bronze_orders():
    return (spark.readStream.format("cloudFiles")
            .option("cloudFiles.format", "json")
            .load("/Volumes/prod/sales/raw/orders/"))

@dp.table
@dp.expect_or_drop("valid_id", "order_id IS NOT NULL")
@dp.expect("positive_amount", "amount > 0")
def silver_orders():
    return (spark.readStream.table("bronze_orders")
            .select("order_id", "customer_id",
                    F.col("amount").cast("decimal(12,2)").alias("amount"), "order_ts"))

@dp.materialized_view
def gold_daily_revenue():
    return (spark.read.table("silver_orders")
            .groupBy(F.to_date("order_ts").alias("day"))
            .agg(F.sum("amount").alias("revenue")))

(On older runtimes the module is import dlt with @dlt.table and @dlt.expect; the ideas are identical.)

You never write the order of execution. silver_orders reads bronze_orders, so Databricks builds the graph, runs bronze first, and runs independent branches in parallel.

flowchart LR
    F["Files in volume"] --> B["bronze_orders (streaming table)"]
    B --> S["silver_orders (streaming table)"]
    S --> G["gold_daily_revenue (materialized view)"]
    S --> C["gold_customer_totals (materialized view)"]

The pipeline graph is inferred from which dataset reads which.

Python Stand-in: resolving a pipeline graph from definitions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Expectations: quality built in

An expectation is a named rule on each row, with a violation policy:

Policy SQL Python Effect
Warn (default) EXPECT (...) @dp.expect Keep the row, count the violation
Drop ON VIOLATION DROP ROW @dp.expect_or_drop Remove the bad row, count it
Fail ON VIOLATION FAIL UPDATE @dp.expect_or_fail Stop the update

The pipeline UI and event log show how many rows passed or failed each rule, which doubles as a data-quality dashboard.

Python Stand-in: expectations with three policies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Change data capture

Source systems send inserts, updates and deletes. AUTO CDC (earlier APPLY CHANGES INTO) turns a feed of changes into a current-state table, ordering by a sequence column so late events do not overwrite newer ones:

SQL
CREATE OR REFRESH STREAMING TABLE customers_current;

CREATE FLOW customers_cdc AS AUTO CDC INTO customers_current
FROM STREAM customers_changes
KEYS (customer_id)
APPLY AS DELETE WHEN op = 'DELETE'
SEQUENCE BY change_ts
STORED AS SCD TYPE 1;

SCD TYPE 2 keeps history with start and end dates instead.

Running and operating a pipeline

  • Create a pipeline (a Pipelines task) pointing at your notebooks or files, with a target catalog and schema, and choose serverless or classic compute.
  • Triggered mode updates once and stops (schedule it as a job task, see Multi-Task Workflows); continuous mode keeps running for low latency.
  • Development mode reuses compute and does not retry, for fast iteration; production mode retries and restarts.
  • A full refresh reprocesses everything from scratch (use carefully, since it rebuilds tables and ignores checkpoints for streaming tables).
  • The event log is a table with updates, expectations and lineage you can query.

When not to use them

Pipelines suit table-to-table ETL. For arbitrary logic, very custom writes (foreachBatch), or one-off scripts, a normal job with notebooks is more flexible.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What do you declare in a declarative pipeline?

    Dependencies are inferred from what each dataset reads.
  2. 2. Which policy removes rows that break a rule?

    DROP ROW (or expect_or_drop) discards the offending rows.
  3. 3. A materialized view is best for:

    Streaming tables ingest; materialized views summarise.
  4. 4. What does SEQUENCE BY do in AUTO CDC?

    The sequence column decides which change is newest.

Summary

  • Streaming tables ingest incrementally; materialized views keep aggregates fresh.
  • The graph is inferred from references; expectations add quality rules with warn, drop or fail.
  • AUTO CDC applies change feeds, with SCD type 1 or 2.
  • Run triggered or continuous, in development or production mode, and read the event log.

Next: Asset Bundles and CI/CD