Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 36 ยท Getting Started

The Lakehouse Architecture

Understand the lakehouse idea: data lake plus warehouse in one system, and the bronze, silver and gold medallion layers used to organise tables in Databricks.

For years companies ran two separate systems. A data lake held raw files of any kind (cheap, flexible, but messy and unreliable). A data warehouse held clean tables for reports (fast and trusted, but expensive and limited to structured data). Teams copied data from lake to warehouse every night, and the copies drifted apart.

The lakehouse is the idea of getting both from one system: files in cheap object storage, plus a table layer on top that adds transactions, schemas and governance. In Databricks that table layer is Delta Lake, and the governance layer is Unity Catalog.

Lake, warehouse, lakehouse

Data lake Data warehouse Lakehouse
Stores Any files Structured tables Any files, with tables on top
Cost Low Higher Low
ACID transactions No Yes Yes (Delta)
BI and SQL Awkward Excellent Good (SQL warehouses)
ML and unstructured data Good Poor Good
Copies of data Many Many One shared copy

"ACID" means a write either fully happens or not at all, readers never see half-written data, and concurrent writers do not corrupt each other. Plain files in a folder have none of that. Delta Lake adds it with a transaction log, which you will meet in The Delta transaction log.

The medallion layers

The most common way to organise tables in a lakehouse is the medallion architecture: three layers with rising quality.

flowchart LR
    S["Sources<br/>files, APIs, databases"] --> B["Bronze<br/>raw, as received"]
    B --> Si["Silver<br/>cleaned, deduplicated, typed"]
    Si --> G["Gold<br/>business aggregates"]
    G --> BI["Dashboards"]
    Si --> ML["ML features"]

Data moves left to right and gets cleaner at each step. Each arrow is a job you can rerun.

  • Bronze keeps the data exactly as it arrived, plus load metadata such as the file name and load time. Nothing is thrown away, so you can always rebuild the later layers.
  • Silver fixes types, removes duplicates and bad rows, and joins reference data. It is the trusted, detailed layer.
  • Gold holds tables shaped for a purpose: daily revenue, customer metrics, features for a model.

This is a convention, not a feature you switch on. In Unity Catalog you might use schemas named bronze, silver and gold:

SQL
CREATE SCHEMA IF NOT EXISTS main.bronze;
CREATE SCHEMA IF NOT EXISTS main.silver;
CREATE SCHEMA IF NOT EXISTS main.gold;

A real pipeline step from bronze to silver looks like this (read-only Databricks code):

Python
from pyspark.sql import functions as F

bronze = spark.read.table("main.bronze.orders_raw")

silver = (bronze
    .dropDuplicates(["order_id"])
    .filter(F.col("amount") > 0)
    .withColumn("order_date", F.to_date("order_ts")))

silver.write.mode("overwrite").saveAsTable("main.silver.orders")

Why layers help

  • Replay. If you find a bug in silver, fix the code and rebuild from bronze.
  • Trust. Consumers read gold or silver and do not need to know the quirks of raw files.
  • Cost and speed. Gold tables are small, so dashboards are quick.
  • Ownership. Different teams can own different layers, with permissions set per schema.

A stand-in medallion pipeline

The stand-in below runs the same three steps on a tiny list of dicts. It is plain Python, not Spark, but each function plays the role of a job.

Python Stand-in: bronze to silver to gold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The next stand-in shows the "replay" benefit. We find a bug in the silver rule (it should also drop orders under 10), fix the function and rebuild everything from the untouched bronze data.

Python Stand-in: replaying from bronze after a fix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Variations and cautions

Some teams add a fourth "platinum" layer or call the layers raw, refined and curated. The names matter less than the rule: raw data is kept, and each layer is reproducible from the previous one. Do not let people edit silver by hand, or you lose that property.

Avoid creating too many layers. Every extra copy costs storage, compute and attention. If a gold table is only a rename of silver, skip it and grant access to silver.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a lakehouse add on top of plain files in object storage?

    The lakehouse keeps cheap storage but adds a table format (Delta Lake) and a catalog so you get warehouse-like reliability.
  2. 2. Which medallion layer keeps data exactly as it arrived?

    Bronze is the raw landing layer. Keeping it unchanged lets you rebuild silver and gold at any time.
  3. 3. Why is it useful that silver can be rebuilt from bronze?

    Because the raw data is preserved, a corrected transformation can simply be run again.
  4. 4. What does ACID guarantee for a write?

    ACID covers atomicity, consistency, isolation and durability, which plain files do not provide.

Summary

  • A lakehouse combines the low-cost storage of a lake with the reliability of a warehouse.
  • Delta Lake supplies the table layer; Unity Catalog supplies governance.
  • The medallion pattern (bronze, silver, gold) organises tables by quality.
  • Keep raw data so every later layer can be rebuilt.

Next: Getting Started with Databricks Free Edition