46. Rebuild a Table from Its Transaction Log

Difficulty: Medium · Topics: Joins, Aggregations, Filtering & Selection

A Delta table's log records every commit as actions: version, action ('add' or 'remove') and the data file path. The table at a version is the set of files whose latest action at or before that version is an add.

as_of holds a single row with the version to read (time travel). Return the path of every live file at that version.

Row order does not matter; column names must match. Your code is graded on 4 test cases, including hidden edge cases.

Sample data

log

version	action	path
0	add	part-000.parquet
0	add	part-001.parquet
1	add	part-002.parquet
2	remove	part-001.parquet
2	add	part-003.parquet
3	remove	part-000.parquet
3	remove	part-002.parquet
3	remove	part-003.parquet
3	add	part-004.parquet

as_of

version
2

Expected output

path
part-000.parquet
part-002.parquet
part-003.parquet

Hints

Hint 1

Keep only log rows with version <= the requested version: cross join the one-row as_of table, or join on a condition.

Hint 2

For each path you need the action of its latest commit. F.max_by("action", "version") does that in one aggregate.

Hint 3

A file can be removed and later added back; only the latest action counts.

Learn the concepts

The Delta transaction log · 8 min read. Commit files, checkpoints and optimistic concurrency: how the _delta_log turns files into a table.
VACUUM and retention · 4 min read. Delete old data files safely without breaking running readers or time travel.
Time travel and RESTORE · 3 min read. Query or roll back to any earlier version of a table, and what retention does to that promise.

PySpark functions you'll practise

groupBy
agg
max_by
crossJoin
filter
select

Browse

Topics: Window Functions · Joins · Aggregations · Pivot, Unpivot & Rollup · Arrays · Null Handling · Conditional Logic · Dates · Filtering & Selection · Strings

Difficulty: Easy · Medium · Hard · PySpark interview roadmap · Learn · All problems