Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 36 ยท Delta Lake

The Delta Transaction Log and ACID

Look inside a Delta table: the _delta_log folder, JSON commits and checkpoints, how ACID and optimistic concurrency work, and how to read DESCRIBE HISTORY.

Every Delta table has a folder called _delta_log. It is the single source of truth about what the table contains. Anyone who reads the table first reads the log to learn which data files are current. Understanding it explains almost every Delta feature: atomic writes, time travel, concurrent writers and why you must never delete files by hand.

What is in the folder

Text
my_table/
  _delta_log/
    00000000000000000000.json
    00000000000000000001.json
    00000000000000000002.json
    ...
    00000000000000000010.checkpoint.parquet
  part-0001-....snappy.parquet
  part-0002-....snappy.parquet

Each numbered JSON file is one commit: a list of actions. Common actions:

Action Meaning
metaData Schema, partition columns, table properties
protocol Minimum reader/writer versions needed
add A data file joins the table (with row statistics: min, max, null counts)
remove A data file leaves the table (it still exists on disk for now)
commitInfo Who, when, which operation and metrics

The current table is the result of replaying all commits in order: start empty, apply every add and remove, and the files left standing are the table. To avoid replaying thousands of small JSON files, Delta writes a checkpoint (a Parquet summary of the state) every so many commits; readers start from the latest checkpoint and replay only newer commits.

A simplified commit file might contain lines like this (read-only illustration):

JSON
{"commitInfo": {"operation": "UPDATE", "operationMetrics": {"numUpdatedRows": "1"}}}
{"remove": {"path": "part-0002.parquet", "dataChange": true}}
{"add": {"path": "part-0004.parquet", "size": 1042, "stats": "{\"numRecords\":2}"}}

ACID, concretely

  • Atomicity: a commit is one new log file. Writers first write the data files, and only when all are finished do they create the next numbered JSON. Readers see either all of a commit or none, so half-finished work stays invisible. The creation of that file is an atomic operation in cloud storage ("put if absent").
  • Consistency: schema and constraints are checked before the commit.
  • Isolation: readers work on a snapshot, the table as of one log version, so a long query never sees changes made after it started.
  • Durability: once the log file exists, the commit is permanent.

Concurrent writers: optimistic concurrency

Two jobs may write at once. Delta lets both prepare their files, and each then tries to create the next log version number:

sequenceDiagram
    participant A as Writer A
    participant L as _delta_log
    participant B as Writer B
    A->>L: read version 5
    B->>L: read version 5
    A->>L: write 000...006.json (succeeds)
    B->>L: write 000...006.json (fails: already exists)
    B->>L: check for conflicts with 006, then retry as 007

Only one writer can create version 6. The other checks whether the winner's changes touch the same files; if not, it retries as version 7, otherwise it fails with a conflict error.

Appends never conflict with each other. Two jobs updating the same rows do. Good partitioning or clustering of the table, and keeping jobs on separate slices of data, reduces conflicts.

Reading the history

SQL
DESCRIBE HISTORY main.sales.orders;

You get one row per version: version, timestamp, userName, operation (WRITE, UPDATE, MERGE, DELETE, OPTIMIZE ...), operationParameters and operationMetrics such as numOutputRows. It is the audit trail for "who changed this table and when".

Stand-in: replaying a log

The stand-in below implements the replay idea: a list of commits with add and remove actions, and a function that returns the active files at any version. This is a simplified model of the protocol.

Python Stand-in: replay commits to find current files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The second stand-in models optimistic concurrency: a writer commits only if the version it started from is still the latest; otherwise it retries after checking for a conflict.

Python Stand-in: optimistic concurrency with retry

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical consequences

  • Never delete or move files inside a Delta folder. The log would still point at them and queries would fail.
  • Many tiny commits (streaming every second) create many log files; checkpoints and compaction keep reading efficient.
  • Table protocol versions: some features (deletion vectors, column mapping) raise the minimum reader version, and older clients can no longer read the table. Upgrade deliberately.
  • Statistics in the log let Delta skip files: if a filter asks for amount > 1000 and a file's max is 900, it is never read ("data skipping").

Common mistakes

  • Running rm on old parquet files to "clean up" instead of using VACUUM.
  • Having several jobs update the same rows concurrently and ignoring conflict errors.
  • Expecting DESCRIBE HISTORY to last forever; log retention is limited by table properties.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How does a reader find which data files are current?

    The log records add and remove actions. Replaying them gives the current file set.
  2. 2. Why do readers never see half-finished writes?

    The new numbered JSON file is created atomically after the data files exist.
  3. 3. Two writers start from version 5. What happens?

    This is optimistic concurrency: only one can create the next version, and the loser rechecks.
  4. 4. What is a checkpoint?

    Checkpoints speed up reading the log by summarising the state at a version.

Summary

  • _delta_log holds numbered JSON commits (plus checkpoints) of add, remove, metaData and commitInfo actions.
  • Replaying the log gives the current table; stopping at an older version gives an older snapshot.
  • ACID comes from writing data files first and then atomically creating the next log file.
  • Concurrent writers use optimistic concurrency and may need to retry or fail on conflicts.

Next: Time Travel and RESTORE