Deletion vectors

What it is

A deletion vector (DV) belongs to one data file and lists the positions of its deleted rows, encoded as a compressed RoaringBitmap. Small DVs can be stored inline in the commit; larger ones go into separate DV files. The add action for a file references its DV, so the log always says "this file, minus these rows".

DELETE FROM events WHERE customer_id = 9001;

Enabling and using

ALTER TABLE events SET TBLPROPERTIES ('delta.enableDeletionVectors' = true);

-- Physically remove soft-deleted rows (e.g. for GDPR erasure), then vacuum
REORG TABLE events APPLY (PURGE);
VACUUM events;

• Supported for DELETE first (Delta 2.4), then UPDATE and MERGE in later Delta 3.x releases.
• Speed-ups are largest when changes touch few rows across many files: point deletes, GDPR requests, small CDC batches.
• Enabling DVs adds a table feature, raising the reader and writer protocol versions; readers must support deletion vectors to read the table at all.

In Iceberg

Iceberg format v2 uses position delete files (file path plus row position) and equality delete files (delete all rows matching these values). Format v3 adds binary deletion vectors stored in Puffin files, with at most one vector per data file, which is much cheaper to read than many position delete files. The two ecosystems converged on the same idea.

Costs

• Reads must load and apply DVs: cheap for a few, noticeable when most files carry large ones.
• Storage holds deleted rows until purge and vacuum, which matters for compliance deadlines.
• Compaction becomes part of the plan, not optional.

Common mistakes