VACUUM and retention

Why old files pile up

Every UPDATE, DELETE, MERGE and OPTIMIZE removes files from the table by writing remove actions, but the files themselves stay on storage. That is what makes time travel and snapshot isolation work. Without cleanup, storage grows forever: a table that is compacted daily can easily hold several times its live size in old files.

What VACUUM deletes

• Files that are not referenced by the current version of the table, and
• were removed (or written by failed jobs) longer ago than the retention threshold: delta.deletedFileRetentionDuration, 7 days by default.
• It never deletes files the current version uses, and it does not touch the transaction log; log cleanup happens separately at checkpoints.

from delta.tables import DeltaTable

DeltaTable.forName(spark, "events").vacuum()        # default retention
DeltaTable.forName(spark, "events").vacuum(240)     # keep 240 hours (10 days)

VACUUM events DRY RUN;           -- list what would be deleted
VACUUM events;                   -- default 7 days
VACUUM events RETAIN 240 HOURS;  -- 10 days

Why the threshold matters

Delta refuses retention below the default unless you disable spark.databricks.delta.retentionDurationCheck.enabled. Keep the threshold longer than your longest running query, your slowest streaming consumer, and the time travel window you promise users.

In Iceberg

-- Remove old snapshots and the files only they reference
CALL catalog.system.expire_snapshots(
  table => 'db.events', older_than => TIMESTAMP '2025-03-01 00:00:00', retain_last => 10);

-- Remove files no metadata references at all (failed writes)
CALL catalog.system.remove_orphan_files(table => 'db.events');

Iceberg splits the job in two: expiring snapshots deletes files that only expired snapshots used; removing orphan files cleans up files no snapshot ever committed. Orphan removal defaults to files older than 3 days, for the same reason as Delta's threshold: an in-progress write's files look like orphans until it commits.

Running it in production

• Schedule it (daily or weekly), after compaction jobs.
• Run DRY RUN the first time and after changing retention.
• VACUUM must list the table's files, which can be slow on huge tables; newer Delta versions offer a lighter mode that uses the log instead of a full listing.
• Coordinate with compliance: deleting a person's rows with DELETE does not remove the data from storage until VACUUM runs after retention.

Common mistakes