47. Files VACUUM Can Delete

Difficulty: Medium · Topics: Joins, Aggregations, Dates, Filtering & Selection

The transaction log has version, day ('YYYY-MM-DD'), action ('add' / 'remove') and path. today holds one row with the current day.

VACUUM may delete a file only if it is not in the current table (its latest action is a remove) and that remove happened at least 7 days before today. Return path and removed_on (the day of that latest remove) for every deletable file.

Row order does not matter; column names must match. Your code is graded on 3 test cases, including hidden edge cases.

Sample data

log

version	day	action	path
0	2025-03-01	add	f1
0	2025-03-01	add	f2
1	2025-03-02	remove	f1
1	2025-03-02	add	f3
2	2025-03-10	remove	f2
2	2025-03-10	add	f4

today

day
2025-03-12

Expected output

path	removed_on
f1	2025-03-02

Hints

Hint 1

Per path, find the latest action and its day: max_by works for both, ordered by version.

Hint 2

Files whose latest action is an add are live and never deletable, even if they were removed earlier.

Hint 3

F.datediff(today_day, removed_on) >= 7; bring the one-row today table in with a cross join.

Learn the concepts

The Delta transaction log · 8 min read. Commit files, checkpoints and optimistic concurrency: how the _delta_log turns files into a table.
VACUUM and retention · 4 min read. Delete old data files safely without breaking running readers or time travel.
max_by / min_by · 4 min read. Return the value from the row where another column is largest or smallest, in one aggregate.

PySpark functions you'll practise

groupBy
agg
max_by
crossJoin
datediff
filter
select

Browse

Topics: Window Functions · Joins · Aggregations · Pivot, Unpivot & Rollup · Arrays · Null Handling · Conditional Logic · Dates · Filtering & Selection · Strings

Difficulty: Easy · Medium · Hard · PySpark interview roadmap · Learn · All problems