Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Projects and Practice

Project: A Mini Lakehouse with Bronze, Silver and Gold

Build a mini lakehouse in Python: land raw JSON, write partitioned Parquet bronze, clean to silver with dedup, aggregate to gold, then compact files and time travel.

This project turns the storage chapter into working code. You build a small medallion lakehouse: bronze (raw, as received), silver (cleaned and deduplicated), gold (aggregates for consumers), using real Parquet files in a temporary folder, partitioned by date. You then compact small files and add a tiny commit log for time travel.

The commit log at the end is a labelled stand-in for what Delta Lake or Iceberg do. The Parquet, partitioning and compaction parts are real pyarrow code.

flowchart LR
  A["Events (JSON lines)"] --> B["Bronze<br/>Parquet, partitioned by ingest date"]
  B --> C["Silver<br/>typed, deduplicated"]
  C --> D["Gold<br/>daily metrics"]

Step 1: simulate an event feed

Python Step 1: generate three days of events

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second day contains a duplicate event, as happens with at-least-once delivery.

Step 2: bronze, partitioned Parquet

Bronze keeps data as received, plus metadata (ingest date, source file). No fixing yet.

Python Step 2: write bronze partitions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

delete_matching replaces a day's partition on rerun, so bronze writes are idempotent as well.

Step 3: silver, clean and deduplicate

Python Step 3: build silver from bronze

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Silver partitions by the event date (when it happened), not the ingest date, which is the right grain for most analytics. Dedup by event_id removed the duplicate delivery.

Step 4: gold, metrics for consumers

Python Step 4: gold daily metrics

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Gold tables are small, well-named, documented and rebuilt from silver whenever the logic changes.

Step 5: small files and compaction

Suppose a micro-batch job wrote a tiny file every few minutes. Compact them.

Python Step 5: compact 30 small files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Thirty files, each carrying its own footer, shrink into one with the same rows and fewer bytes.

Step 6: stand-in commit log and time travel

A real table format would record each write in a transaction log. This stand-in keeps just a list of versions, each naming its files.

Python Step 6: versions, rollback and time travel (stand-in)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Extensions

  1. Add a data quality gate between bronze and silver that quarantines events with a negative amount.
  2. Add late events: write events for 2026-03-01 in the 2026-03-03 feed and check that silver partitions by event date handle them.
  3. Compute a rolling 7-day gold metric with a window.
  4. Add an ingest_date filter so each run processes only new bronze partitions (incremental).
  5. Estimate the bytes scanned by a query with and without partition filters.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What belongs in the bronze layer?

    Bronze is the replayable source of truth for later layers.
  2. 2. Why partition silver by event date instead of ingest date?

    Event time answers business questions; ingest time just tracks when data arrived.
  3. 3. What problem does compaction solve?

    Compaction rewrites many small files as fewer large ones.
  4. 4. In the stand-in, why is a rollback implemented as a new commit?

    Table formats never edit history; they append a new version that reverses a change.

Summary

  • Medallion layers: bronze (raw), silver (clean, deduplicated), gold (consumer metrics).
  • Partition for how data is queried, write partitions idempotently, compact small files.
  • A commit log over immutable files gives atomic changes, rollback and time travel.

Next: Practice Exercises and Interview Questions