27. Forward-Fill Missing Readings
Difficulty: Hard · Topics: Window Functions
IoT sensors sometimes send a reading with no temperature. Replace each missing temp with the most recent earlier reading from the same sensor.
A sensor's leading readings before its first real value stay null. Return sensor_id, ts, temp.
Row order does not matter; column names must match. Your code is graded on 3 test cases, including hidden edge cases.
Sample data
readings
| sensor_id | ts | temp |
|---|---|---|
| s1 | 2024-01-01 00:00:00 | 20.5 |
| s1 | 2024-01-01 01:00:00 | null |
| s1 | 2024-01-01 02:00:00 | null |
| s1 | 2024-01-01 03:00:00 | 22 |
| s2 | 2024-01-01 00:00:00 | 18 |
| s2 | 2024-01-01 01:00:00 | null |
Expected output
| sensor_id | ts | temp |
|---|---|---|
| s1 | 2024-01-01 00:00:00 | 20.5 |
| s1 | 2024-01-01 01:00:00 | 20.5 |
| s1 | 2024-01-01 02:00:00 | 20.5 |
| s1 | 2024-01-01 03:00:00 | 22 |
| s2 | 2024-01-01 00:00:00 | 18 |
| s2 | 2024-01-01 01:00:00 | 18 |
Hints
Hint 1
Partition by sensor, order by timestamp, and look at all rows from the start up to the current one.Hint 2
F.last("temp", ignorenulls=True) over that frame returns the latest non-null value.Hint 3
The frame is.rowsBetween(Window.unboundedPreceding, Window.currentRow).PySpark functions you'll practise
- Window spec
- Rows Between
- orderBy
Related problems
- 7-Day Moving Average · Medium · Window Functions
- Latest Order per Customer · Medium · Window Functions
- Running Total by Region · Medium · Window Functions
- Top Two Salary Levels per Department · Medium · Window Functions
- Month-over-Month Change · Medium · Window Functions
Browse
Topics: Window Functions · Joins · Aggregations · Pivot, Unpivot & Rollup · Arrays · Null Handling · Conditional Logic · Dates · Filtering & Selection · Strings
Difficulty: Easy · Medium · Hard · PySpark interview roadmap · All problems