Skip to content
Tutorials Center
My learning

Topics

Lesson 22 of 36 ยท Storage Formats and Lakehouses

Lakehouse Table Formats: Delta, Iceberg, Hudi

Understand how Delta Lake, Apache Iceberg and Apache Hudi add transactions, snapshots, time travel and schema evolution to files, with a runnable mini model.

Parquet files on cheap object storage are fast to scan, but they are just files. Nobody guarantees that two jobs writing at once will not corrupt the table, that a reader will not see half of an overwrite, or that you can undo a bad load. A table format is a thin metadata layer on top of the files that turns a folder of Parquet into a real table with transactions. A storage layer built this way, with warehouse-like reliability on lake-like storage, is called a lakehouse.

Delta Lake, Iceberg and Hudi need Spark, Trino, Flink or similar engines, which cannot run in a browser tab. The code in this lesson is therefore a small, clearly labelled stand-in that models the core idea (a commit log pointing to immutable files) in plain Python. The real products are described in text and read-only snippets.

The problem with plain files

Imagine a job that replaces the data of sales:

  1. It deletes the old files.
  2. It writes the new files.

A dashboard that queries between steps 1 and 2 sees an empty table. A crash after step 1 loses the table. Two jobs appending concurrently may overwrite each other's file names. Plain files give you no atomicity, no isolation and no history.

The core idea: immutable files plus a commit log

All three formats share one design:

  • Data files are immutable. They are never edited, only added or logically removed.
  • A metadata log lists which files make up the table at each version.
  • A commit is an atomic action: the new version of the log either appears completely or not at all (using an atomic file create, a compare-and-swap on a pointer, or a catalog).
  • Readers pick one version and read exactly the files it lists, so they never see a half-finished write (snapshot isolation).
flowchart LR
  V0["Version 0<br/>files: a"] --> V1["Version 1<br/>files: a, b"]
  V1 --> V2["Version 2<br/>files: b, c<br/>(a removed)"]
  R["Reader at version 1"] -.-> V1

Old files stay in storage until a cleanup job removes them, which is what makes time travel (reading an earlier version) possible.

A mini table format

Python Stand-in: a commit log over immutable files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is exactly why updates and deletes on a lake are possible: they are file rewrites recorded in one commit. Real formats add delete vectors or position-delete files to avoid rewriting whole files for tiny changes.

Concurrency: optimistic commits

Python Two writers, one winner

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real formats use the same optimistic concurrency control: writers prepare their files without locking, then try an atomic commit; if the table moved on, they check whether their change still applies (no overlapping files) and retry.

The three formats compared

Feature Delta Lake Apache Iceberg Apache Hudi
Metadata JSON log plus checkpoints in _delta_log/ Metadata JSON, manifest lists, manifests Timeline in .hoodie/
Strong point Tight Spark/Databricks integration Engine-neutral spec, hidden partitioning, partition evolution Record-level upserts and streaming ingestion
Updates Copy-on-write, deletion vectors Copy-on-write or merge-on-read Copy-on-write or merge-on-read
Time travel By version or timestamp By snapshot id or timestamp By commit instant
Catalog Unity Catalog, Hive metastore, files REST, Glue, Hive, Nessie, Polaris Hive, Glue, others

A real Delta example looks like this (read-only; it needs Spark):

SQL
-- Spark SQL with Delta Lake
CREATE TABLE sales (id BIGINT, amount DECIMAL(10,2), sale_date DATE)
USING DELTA PARTITIONED BY (sale_date);

MERGE INTO sales t USING updates u ON t.id = u.id
WHEN MATCHED THEN UPDATE SET t.amount = u.amount
WHEN NOT MATCHED THEN INSERT *;

SELECT * FROM sales VERSION AS OF 3;   -- time travel

What a table format gives you

  • ACID transactions on object storage.
  • Time travel and rollback to fix bad loads.
  • Schema evolution tracked centrally (add, rename and drop columns using column ids, not names).
  • Upserts and deletes (MERGE), important for change data capture and GDPR deletion.
  • Hidden partitioning and partition evolution in Iceberg: queries filter by the real column and the format maps it to partitions.
  • Maintenance commands: compaction, expire old snapshots, remove orphan files.

Choosing

Pick the format your main engine and catalog support best. Iceberg has the widest multi-engine support; Delta is the natural choice on Databricks and increasingly readable by other engines; Hudi shines for record-level streaming upserts. Conversion layers exist (such as metadata translation between Delta and Iceberg), so the decision is less permanent than it once was.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What makes a commit atomic in a table format?

    Data files are immutable; the single step of publishing the new log version decides what readers see.
  2. 2. How is an UPDATE implemented on immutable Parquet files?

    Copy-on-write rewrites the affected file; merge-on-read stores deltas to apply at read time.
  3. 3. Two writers start from version 5 and both try to commit. What happens?

    Optimistic concurrency: the atomic commit fails for the writer whose base version is stale.
  4. 4. Why can you read an older version of a table (time travel)?

    Immutable files plus a versioned log mean an old snapshot's files still exist until expired.

Summary

  • Table formats add a metadata log over immutable files to give ACID, snapshots, time travel and safe updates.
  • Writers use optimistic concurrency; readers see a consistent snapshot.
  • Delta, Iceberg and Hudi share the idea but differ in metadata layout, engines and update strategy.
  • Run maintenance: compact files, expire snapshots and remove orphan files.

Next: Dimensional Modeling and the Star Schema