42. Combine Batches with Schema Drift

Difficulty: Medium · Topics: Filtering & Selection

Two yearly extracts must be combined into one table. The 2024 extract has its columns in a different order and a new currency column that 2023 lacks.

Return all rows with columns id, name, amount, currency (in that order); 2023 rows have null currency.

Row order does not matter; column names must match. Your code is graded on 3 test cases, including hidden edge cases.

Sample data

batch_2023

id	name	amount
1	Ana	10
2	Bo	20

batch_2024

amount	name	id	currency
30	Cy	3	INR
40	Di	4	USD

Expected output

id	name	amount	currency
1	Ana	10	null
2	Bo	20	null
3	Cy	30	INR
4	Di	40	USD

Hints

Hint 1

union matches columns by position, which would mix up name and amount here.

Hint 2

unionByName matches by name; allowMissingColumns=True fills absent columns with null.

Hint 3

Start from the 2023 table so its column order comes first.

PySpark functions you'll practise

• unionByName

Related problems

• Upsert: Merge New Records into a Table · Hard · Window Functions
• Filter High Earners · Easy · Filtering & Selection
• Add a Bonus Column · Easy · Filtering & Selection
• Distinct User Actions · Easy · Filtering & Selection
• Categorize Orders · Easy · Conditional Logic

Browse

Topics: Window Functions · Joins · Aggregations · Pivot, Unpivot & Rollup · Arrays · Null Handling · Conditional Logic · Dates · Filtering & Selection · Strings

Difficulty: Easy · Medium · Hard · PySpark interview roadmap · All problems