The Delta transaction log

Why it exists

Picture a plain Parquet table: a folder, and "the table" is every file in it. That definition breaks in three ways.

1. Half-finished writes are visible. A job writing 100 files that dies after 60 leaves 60 files behind. Readers see a partial result with no error.
2. Updates are not atomic. Rewriting a file means deleting the old one and writing a new one. A reader between those two steps sees missing or duplicate rows.
3. Listing is slow. Listing millions of objects in S3 or ADLS takes minutes, and engines must trust that the listing is complete.

Delta changes the definition: the table is the set of files the log says are live, not whatever happens to be in the folder.

The mental model

Think of a bank ledger. The ledger never stores your balance. It stores every deposit and withdrawal, and the balance is what you get by adding them up in order. You can also get the balance as of any past date by stopping early.

Inside _delta_log

Next to the data files sits one folder. Each commit is a JSON file named by its version, padded to 20 digits. Every few commits Delta also writes a Parquet checkpoint.

events/
├── part-00000-….snappy.parquet
├── part-00001-….snappy.parquet
└── _delta_log/
    ├── 00000000000000000000.json      # version 0
    ├── 00000000000000000001.json      # version 1
    ├── …
    ├── 00000000000000000010.checkpoint.parquet  # full state at v10
    └── _last_checkpoint               # points to newest checkpoint

Each line of a commit file is one action:

Walk through four commits

Step through what happens to the log and the files as an events table changes. Watch the difference between the files in the table and the files on storage.

df.write.format("delta").saveAsTable("events")

CREATE TABLE events USING delta
AS SELECT * FROM staging;

{"protocol":{"minReaderVersion":1,"minWriterVersion":2}}
{"metaData":{"schemaString":"…","partitionColumns":[]}}
{"add":{"path":"part-000.parquet","dataChange":true,"stats":"{\"numRecords\":5000,…}"}}
{"add":{"path":"part-001.parquet","dataChange":true,…}}
{"commitInfo":{"operation":"CREATE TABLE AS SELECT",…}}

new_rows.write.format("delta").mode("append").saveAsTable("events")

INSERT INTO events SELECT * FROM staging_day2;

{"add":{"path":"part-002.parquet","dataChange":true,…}}
{"commitInfo":{"operation":"WRITE","operationParameters":{"mode":"Append"},…}}

DeltaTable.forName(spark, "events").delete("country = 'FR'")

DELETE FROM events WHERE country = 'FR';

{"remove":{"path":"part-001.parquet","dataChange":true,"deletionTimestamp":…}}
{"add":{"path":"part-003.parquet","dataChange":true,…}}
{"commitInfo":{"operation":"DELETE",…}}

DeltaTable.forName(spark, "events").optimize().executeCompaction()

OPTIMIZE events;

{"remove":{"path":"part-000.parquet","dataChange":false,…}}
{"remove":{"path":"part-002.parquet","dataChange":false,…}}
{"remove":{"path":"part-003.parquet","dataChange":false,…}}
{"add":{"path":"part-004.parquet","dataChange":false,…}}
{"commitInfo":{"operation":"OPTIMIZE",…}}

How a read works

Replaying every commit since version 0 would get slower forever. Checkpoints cap the work. When a query starts, the reader:

1. 1
   Reads _last_checkpoint to find the newest checkpoint, say version 40.
2. 2
   Loads that checkpoint: the full list of live files at version 40.
3. 3
   Lists the log for JSON commits after 40, and applies 41, 42, … in order.
4. 4
   Uses the min and max statistics on each add to skip files that cannot match the query's filter, then reads only the rest.

Step 4 is why Delta often reads far less data than a plain Parquet folder: file skipping uses statistics already in the log, with no listing and no opening of file footers.

How writers avoid conflicts

Delta uses optimistic concurrency: writers do not lock the table. They assume nobody else is writing, and check at the end.

1. Read the table at the latest version, say 7.
2. Write new Parquet files. Nobody can see them yet, because no commit lists them.
3. Try to create …08.json with a "create only if it does not exist" write.
4. If another writer created version 8 first, check whether its changes conflict with ours. No conflict: retry as version 9. Conflict, such as both deleting from the same files: fail with a concurrent modification exception.

Step 3 is the whole trick. Atomicity comes from the storage system guaranteeing that only one writer can create a given file name. It is also why a crash after step 2 is harmless: the orphaned files are never referenced, and VACUUM cleans them up later.

Time travel

Because old commits stay in the log and removed files stay on storage, reading an older version just means replaying fewer commits.

# Read version 1
spark.read.option("versionAsOf", 1).table("events")

# See every commit
DeltaTable.forName(spark, "events").history().show()

# Roll the table back (this writes a new commit)
DeltaTable.forName(spark, "events").restoreToVersion(1)

-- Read version 1
SELECT * FROM events VERSION AS OF 1;

-- See every commit
DESCRIBE HISTORY events;

-- Roll the table back (this writes a new commit)
RESTORE TABLE events TO VERSION AS OF 1;

Retention: two clocks

Time travel needs two things to still exist: the log entries and the data files. Each has its own retention setting, and the shorter one wins.

With the defaults, you have 30 days of history in the log but, after a VACUUM, only about 7 days of versions you can actually read.

The same idea in Iceberg and Hudi

Common mistakes