Skip to content
Tutorials Center
My learning

Topics

Lesson 37 of 47 ยท Architecture and Operations

Data Lakes, Warehouses and Lakehouses

Compare data lakes, warehouses and lakehouses: zones, schema-on-read, Delta, Iceberg and Hudi table formats, and a runnable versioned table with snapshots.

As companies collected more data than their databases could hold, two big ideas appeared for storing it for analysis. The data warehouse keeps clean, modeled tables in a database built for queries. The data lake keeps everything, in raw form, on cheap storage such as HDFS or a cloud object store, and decides later what it means. The newest idea, the lakehouse, tries to give lake storage the reliability of a warehouse.

This lesson compares the three, explains lake zones and schema-on-read, and shows what open table formats (Delta Lake, Apache Iceberg, Apache Hudi) add: transactions, snapshots and time travel. It follows Spark, Flink and What Came After MapReduce.

The real systems cannot run on the site. Their commands are read-only. The runnable examples are Python stand-ins, including a tiny versioned table with snapshots that mirrors how table formats work internally.

Warehouse versus lake

A data warehouse (Teradata, Snowflake, BigQuery, Redshift) stores structured data in tables with a fixed schema. Data is cleaned and modeled before it is loaded (schema-on-write), queries are fast and consistent, and users trust the numbers. It is usually more expensive per terabyte and less comfortable with unstructured data such as images, logs or free text.

A data lake stores files of any kind, cheaply, in their original format. Structure is applied when the data is read (schema-on-read): the same raw JSON can be interpreted differently by different jobs. This is flexible and good for data science and machine learning, but it risks becoming a data swamp: unlabeled files nobody understands or trusts.

Warehouse vs lake

Warehouse Modeled tables, schema-on-write, trusted SQL.
Lake Any file, cheap storage, schema-on-read.
Lakehouse Lake storage with table guarantees.

The lakehouse tries to be both.

1 / 2

Schema-on-read in practice means that the file stays raw and each reader declares columns. A Hive external table is exactly that: a schema laid over files that already exist.

SQL
-- read-only HiveQL: schema applied on read, files untouched
CREATE EXTERNAL TABLE raw_events (
  user_id STRING, action STRING, ts STRING
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
LOCATION '/lake/raw/events/';

Zones: bringing order to a lake

Well-run lakes are divided into zones, usually folders or buckets with different rules. Names vary; the common pattern is a "medallion" of three layers:

  • Raw (bronze): data exactly as it arrived, immutable, kept for replay and audit. Land it here first, as recommended in Getting Data In.
  • Curated (silver): cleaned, deduplicated, typed, conformed to shared definitions, in a columnar format such as Parquet and sensible partitions.
  • Consumption (gold): business-level tables and aggregates built for dashboards and reports.
flowchart LR
    S["Sources"] --> R["Raw zone<br/>(as received)"]
    R --> C["Curated zone<br/>(clean, typed)"]
    C --> G["Gold zone<br/>(business tables)"]
    G --> B["BI and reports"]
    C --> M["ML and data science"]

Data gets cleaner and more valuable as it moves right.

A catalog (Hive Metastore, AWS Glue, Unity Catalog) registers tables, schemas, owners and locations, so files become discoverable tables. Without one a lake turns into a swamp.

Object stores replace HDFS

Many modern lakes sit on cloud object stores (Amazon S3, Azure Data Lake Storage, Google Cloud Storage) instead of HDFS. They are cheap, almost infinitely large, and storage is separated from compute, so you can start a Spark cluster only when needed and shut it down after. Differences you must remember:

  • Objects are immutable whole blobs; there is no append and no cheap in-place edit.
  • Rename is not atomic (it is a copy plus delete), which breaks the old HDFS trick of writing to a temp directory and renaming to commit. Table formats fix this.
  • Listing many files is slow, so the small-files problem returns in a different form.
  • There is no data locality: bytes cross the network to compute.

The problem with plain files

A table made of loose Parquet files has weak points:

  • A job that crashes halfway leaves half-written files visible. Readers see partial results.
  • Two writers at once can corrupt each other.
  • You cannot update or delete one row; you rewrite files by hand (the GDPR delete problem).
  • There is no way to ask "what did this table look like yesterday?".
  • Schema changes break old files silently.

Open table formats

Delta Lake, Apache Iceberg and Apache Hudi add a metadata layer over data files. The data stays in Parquet, but a transaction log or manifest tree decides which files belong to the table at each version. All three provide:

  • ACID transactions: a commit is a single atomic step, so readers see either the old or the new version, never half.
  • Snapshots and time travel: every commit creates a new snapshot; you can query an older one.
  • Row-level updates, deletes and merges (MERGE INTO) done by rewriting only affected files or by writing delta files.
  • Schema evolution tracked in metadata.
  • Optimizations: file statistics for pruning, compaction of small files, partition evolution (Iceberg).

How they track state differs in detail. Delta keeps an ordered JSON transaction log (_delta_log/000...0012.json) with add-file and remove-file actions plus periodic checkpoints. Iceberg keeps a tree: a table metadata file points to a snapshot, which points to manifest lists, then manifests, then data files. Hudi is built around upserts with commit timelines and index structures, and was designed for incremental ingestion.

A read-only Spark SQL example, time travel included:

SQL
-- read-only, Iceberg or Delta style
CREATE TABLE lake.curated.orders (order_id BIGINT, status STRING, amount DOUBLE)
USING iceberg PARTITIONED BY (days(order_ts));

MERGE INTO lake.curated.orders t
USING staging.orders_changes s ON t.order_id = s.order_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

SELECT * FROM lake.curated.orders TIMESTAMP AS OF '2026-01-01 00:00:00';
DELETE FROM lake.curated.orders WHERE customer_id = 4711;   -- GDPR request

The key trick is that an update never changes a file. It writes new files and commits a new version of the file list, so old versions remain valid until cleaned up (VACUUM in Delta, expire_snapshots in Iceberg). The stand-in below builds exactly that.

Python A tiny versioned table with snapshots and time travel

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the output: version 2 lost row 2, yet version 1 still shows it, because the old file part-001.parquet still exists. Only part-001 was rewritten (as part-003), while part-002 was reused untouched. Cleaning up files that no snapshot references is what VACUUM does.

The second stand-in demonstrates why atomic commits matter: a failing writer leaves readers on the last good snapshot.

Python Atomic commit: a crashed writer never exposes partial data

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Which one should you choose?

  • Warehouse: mostly structured data, SQL users, strong governance, and you accept cost and some lock-in for simplicity.
  • Lake: large volumes of varied data, machine learning, cheapest storage, with teams able to manage quality themselves.
  • Lakehouse: you want one copy of the data on open files, with transactions, BI speed and ML access. Pick a table format that your engines support (Iceberg has the widest engine support, Delta is tightly tied to Spark and Databricks though also open, Hudi shines for near-real-time upserts).
  • Hybrid is common: a lakehouse for most data and a warehouse for the most curated, performance-critical marts.

Whatever the choice, the basics decide success: zones, a catalog, ownership, and quality checks. The next lesson covers security, governance and data quality.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is schema-on-read?

    A lake keeps raw files; each reader (for example a Hive external table) declares how to interpret them.
  2. 2. What is the purpose of the raw zone in a lake?

    Keeping the original data lets you reprocess when logic changes or bugs are found.
  3. 3. Why do object stores break the old 'write to a temp folder, then rename' commit?

    On S3-like stores a rename copies objects one by one, so a crash can leave partial results. Table formats commit by writing one new metadata version.
  4. 4. How does a table format implement a row-level DELETE on immutable Parquet files?

    Files stay immutable. The new snapshot lists the rewritten files, and older snapshots remain readable until cleaned.

Summary

  • Warehouses use schema-on-write for trusted, modeled tables; lakes store raw files cheaply with schema-on-read; lakehouses add table guarantees on lake storage.
  • Zones (raw, curated, gold) and a catalog keep a lake from becoming a swamp.
  • Object stores are cheap and separate storage from compute, but have immutable objects and non-atomic rename.
  • Delta, Iceberg and Hudi add ACID commits, snapshots and time travel, row-level changes and schema evolution through a metadata layer over Parquet files.

Next: Security, Governance and Data Quality.