48. Parse Application Log Lines
Difficulty: Medium · Topics: Strings, Filtering & Selection
Each line in logs looks like 2024-03-01 12:00:01 ERROR [payment] Card declined: date, time, level, service in square brackets, then a free-text message.
Return ts (date and time), level, service and message for every line that matches this format; skip malformed lines.
Row order does not matter; column names must match. Your code is graded on 3 test cases, including hidden edge cases.
Sample data
logs
| line |
|---|
| 2024-03-01 12:00:01 ERROR [payment] Card declined |
| 2024-03-01 12:00:05 INFO [auth] User 42 logged in |
| garbage line |
Expected output
| ts | level | service | message |
|---|---|---|---|
| 2024-03-01 12:00:01 | ERROR | payment | Card declined |
| 2024-03-01 12:00:05 | INFO | auth | User 42 logged in |
Hints
Hint 1
One regular expression with capture groups:^(\S+ \S+) (\w+) \[([^\]]+)\] (.*)$.Hint 2
F.regexp_extract(col, pattern, n) returns group n, or "" when the line doesn't match.Hint 3
Filter out rows where the extracted level is empty.PySpark functions you'll practise
- filter
- select
- regexp_extract
Related problems
- Word Count · Medium · Arrays
- Filter High Earners · Easy · Filtering & Selection
- Latest Order per Customer · Medium · Window Functions
- Top Two Salary Levels per Department · Medium · Window Functions
- Three-Day Login Streak · Hard · Window Functions
Browse
Topics: Window Functions · Joins · Aggregations · Pivot, Unpivot & Rollup · Arrays · Null Handling · Conditional Logic · Dates · Filtering & Selection · Strings
Difficulty: Easy · Medium · Hard · PySpark interview roadmap · All problems