45. Iceberg Hidden Partition Layout
Difficulty: Medium · Topics: Aggregations, Dates
An Iceberg table of events (event_id, user_id, ts as 'YYYY-MM-DD HH:MM:SS') is partitioned by day(ts) and bucket(4, user_id). Users never write partition values; Iceberg derives them.
Work out the layout the data would have. Use event_day = the date of ts as 'YYYY-MM-DD' and, as a simplified bucket transform, bucket = user_id modulo 4. Return event_day, bucket and events (rows in that partition). Rows with a null ts or user_id go to a partition whose value is null, as in Iceberg.
Row order does not matter; column names must match. Your code is graded on 3 test cases, including hidden edge cases.
Sample data
events
| event_id | user_id | ts |
|---|---|---|
| 1 | 10 | 2025-03-01 09:15:00 |
| 2 | 11 | 2025-03-01 10:00:00 |
| 3 | 14 | 2025-03-01 23:59:59 |
| 4 | 10 | 2025-03-02 00:00:01 |
| 5 | 7 | 2025-03-02 12:30:00 |
Expected output
| event_day | bucket | events |
|---|---|---|
| 2025-03-01 | 2 | 2 |
| 2025-03-01 | 3 | 1 |
| 2025-03-02 | 2 | 1 |
| 2025-03-02 | 3 | 1 |
Hints
Hint 1
F.date_format("ts", "yyyy-MM-dd") gives the day; it returns null for a null timestamp.Hint 2
F.col("user_id") % 4 is the simplified bucket; null stays null.Hint 3
Group by both derived columns and count.Learn the concepts
- Hidden partitioning and partition evolution · 3 min read. Iceberg partition transforms, and changing a table layout without rewriting its data.
- Partitioning done right · 4 min read. When to partition a table, how to choose the column, and how over-partitioning backfires.
- trunc / add_months · 3 min read. Truncate dates to the month or year and shift them by whole months, including month-end rules.
PySpark functions you'll practise
- groupBy
- agg
- count
- date_format
- withColumn
Related problems
- Monthly Retention by Cohort · Hard · Joins
- Three-Day Login Streak · Hard · Window Functions
- Files VACUUM Can Delete · Medium · Joins
- Fill Missing Dates (Calendar Spine) · Hard · Joins
- Consecutive Status Runs (Gaps and Islands) · Hard · Window Functions
Browse
Topics: Window Functions · Joins · Aggregations · Pivot, Unpivot & Rollup · Arrays · Null Handling · Conditional Logic · Dates · Filtering & Selection · Strings
Difficulty: Easy · Medium · Hard · PySpark interview roadmap · Learn · All problems