Change Data Feed

Why it exists

Downstream tables built from a source table usually need only the rows that changed since the last run. Without row-level changes, you either reprocess everything or compare snapshots, both expensive. Reading appended files alone misses updates and deletes. Change Data Feed (CDF) gives you exact row changes.

Enabling it

ALTER TABLE customers SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

-- or at creation
CREATE TABLE customers (...) TBLPROPERTIES (delta.enableChangeDataFeed = true);

Changes are recorded only from the version where it was enabled onward. For inserts and whole-file deletes, Delta derives changes from the log; for updates, merges and partial deletes, it writes extra change files in a _change_data folder.

What you get back

Reading changes

# Batch: changes in versions 5 through 10
changes = (spark.read.option("readChangeFeed", "true")
    .option("startingVersion", 5).option("endingVersion", 10)
    .table("customers"))

# Streaming: continuously, from where the checkpoint left off
stream = (spark.readStream.option("readChangeFeed", "true")
    .option("startingVersion", 5).table("customers"))

SELECT * FROM table_changes('customers', 5, 10);
SELECT * FROM table_changes('customers', '2025-03-01 00:00:00');

Using it downstream

1. 1
   Read changes since the last processed version (stored in your job state, or tracked by a streaming checkpoint).
2. 2
   Drop update_preimage rows unless you need them (for example to subtract old values from aggregates).
3. 3
   Keep the latest change per key within the batch: a row can change several times between runs.
4. 4
   MERGE the result into the target: delete on delete, upsert otherwise.

Iceberg offers a similar capability through changelog views (create_changelog_view procedure in Spark), computed from snapshots.

Common mistakes