Skip to content
Great engineers know 3 min read · All formats

Concurrency and conflicts

Optimistic concurrency, the conflicts that fail a commit, and how to design writers around them.

On this page

You will learn

  • How optimistic concurrency works in table formats
  • Which concurrent operations conflict and which never do
  • The isolation levels Delta offers
  • How to design pipelines that avoid conflicts

Read first

Comfortable with these? Read on.

TL;DR Writers do not lock tables. Each prepares its changes and commits against the version it started from; if someone committed in between, the format checks whether the two changes overlap. Blind appends never conflict; two operations that read and rewrite the same files do.

Optimistic concurrency

  1. 1
    Record the table version you start from (the read snapshot).
  2. 2
    Read what you need and write new data files.
  3. 3
    Try to commit as the next version.
  4. 4
    If another commit landed first, compare: did it change files you read or rewrote, or the schema? If not, rebase and commit after it. If yes, fail with a conflict error and let the job retry.

It works well because most concurrent operations touch different data. It works badly when many writers update the same files at the same time.

What conflicts

Your operation vs a concurrent...Append (INSERT)UPDATE / DELETE / MERGEOPTIMIZE
Append (INSERT)Never conflictsNever conflictsNever conflicts
UPDATE / DELETE / MERGECan conflict, if the append added files in the data you readCan conflict, if both touch the same filesCan conflict, if both rewrite the same files
OPTIMIZENever conflictsCan conflictCan conflict

This follows Delta's documented behaviour under the default WriteSerializable isolation; Iceberg applies similar validation (for example, a MERGE checks that no conflicting files were added or deleted in the data it read).

Delta conflict errors

ExceptionTypical cause
ConcurrentAppendExceptionFiles were added to the data your operation read, e.g. an append into the partition you are updating
ConcurrentDeleteReadExceptionA file you read was deleted or rewritten by another commit
ConcurrentDeleteDeleteExceptionTwo operations deleted or rewrote the same file
MetadataChangedExceptionSchema or table properties changed concurrently
ProtocolChangedExceptionThe table protocol was upgraded concurrently

Isolation levels

WriteSerializable (default)

Writes are serializable among themselves; reads may see a state that does not correspond to a serial order with concurrent appends. Lets appends proceed alongside other operations.

Serializable

Strictest: reads and writes are serializable. More operations conflict, including appends with concurrent reads of the same data.

Designing around conflicts

  • Make conditions explicit and disjoint. A MERGE with ON t.id = s.id AND t.event_date = '2025-03-14' only reads that date's files, so it no longer conflicts with a job writing another date.
  • Partition or cluster by the dimension writers split on, so their file sets do not overlap.
  • Prefer appends for ingestion, and apply updates in a single downstream job.
  • Schedule maintenance (OPTIMIZE, compaction) away from heavy update windows, or limit it to cold partitions.
  • Retry on conflict exceptions with backoff; the operation is safe to repeat if it is idempotent.
Note: Databricks adds finer-grained conflict detection (row-level concurrency with deletion vectors) that reduces conflicts between operations touching different rows of the same files. Open-source behaviour is file-level.

Common mistakes

Two jobs MERGE into the same table without disjoint conditions

They read overlapping files and conflict repeatedly.

Running OPTIMIZE during heavy updates

Both rewrite the same files.

Not retrying

Conflicts are expected under concurrency; retry idempotent operations.

Key takeaways

  • Writers commit optimistically and check for overlapping changes at commit time.
  • Blind appends never conflict; operations rewriting the same files do.
  • Delta defaults to WriteSerializable; Serializable is stricter.
  • Use disjoint, explicit predicates, aligned layouts and retries.

Check yourself

3 questions

1. Can two concurrent blind appends to a Delta table conflict?

Show the answer

No. Appends add new files without reading existing data, so they never conflict.

2. How do you stop two MERGE jobs on different dates from conflicting?

Show the answer

Include the date in each MERGE condition so they read disjoint files. Disjoint predicates mean disjoint read sets.

3. What is Delta's default isolation level?

Show the answer

WriteSerializable. WriteSerializable is the default.

Go deeper

Lakehouse
The Delta transaction logLakehouse
MERGE INTO and upsertsLakehouse
ACID on object storage

Primary sources: Delta: concurrency control · Iceberg: reliability

← Previous
Liquid clustering Next →
Catalogs: Unity, Polaris, Iceberg REST