posexplode
Explode an array and keep each element position, plus the _outer variant that keeps empty arrays.
On this page
Show code in
Every code block on the page follows this.
You will learn
- How posexplode returns each element with its position
- How to keep rows with empty arrays using posexplode_outer
- How to explode two arrays in step
- How to rebuild an array in order afterwards
F.posexplode(arr) emits one row per element with two columns, pos (0-based) and col. Like explode it drops empty and null arrays; posexplode_outer keeps them.What it does
An array has an order: the third stop of a trip means something. explode throws that order away. posexplode keeps it as a column, so you can filter, join or sort on the position.
Step by step
routes
| trip | stops |
|---|---|
| T1 | [Pune, Lonavala, Mumbai] |
| T2 | [Delhi, Agra] |
| T3 | [] |
posexplode
| trip | pos | stop |
|---|---|---|
| T1 | 0 | Pune |
| T1 | 1 | Lonavala |
| T1 | 2 | Mumbai |
| T2 | 0 | Delhi |
| T2 | 1 | Agra |
Positions start at 0. T3 disappears because its array is empty.
Run the example
from pyspark.sql import functions as F result = routes.select("trip", F.posexplode_outer("stops").alias("pos", "stop"))
SELECT trip, pos, stop FROM routes LATERAL VIEW OUTER posexplode(stops) t AS pos, stop
Switch to PySpark to edit and run this example in your browser.
This uses the _outer variant, so T3 survives with null pos and stop. Note alias with two names: a generator that outputs two columns takes one alias per column.
What the position is good for
- First, last or nth element as rows: filter
pos == 0. For a single value,element_at(arr, 1)(1-based) orarr[0]is simpler. - Consecutive pairs: posexplode, then self-join on
a.pos + 1 = b.posto get each (stop, next stop) leg of a trip. - Parallel arrays: two arrays where element i of one belongs with element i of the other. Explode one with positions and pick from the other with
element_at(other, pos + 1), or zip them first witharrays_zipand explode once.
Putting an array back together in order
After transforming exploded rows, rebuild each array in its original order by collecting (pos, value) structs and sorting them: F.sort_array(F.collect_list(F.struct("pos", "stop"))), then take the stop field. A plain collect_list does not keep the order.
F.transform("stops", lambda s: F.upper(s)) keeps the array and its order with no shuffle and no regrouping.Common mistakes
Assuming positions start at 1
Losing empty arrays
Exploding two arrays separately
Key takeaways
- posexplode returns each element with its 0-based position.
- posexplode_outer keeps empty and null arrays.
- Positions enable nth-element filters, consecutive pairs and parallel arrays.
- Rebuild ordered arrays with sort_array over (pos, value) structs.
Check yourself
3 questions1. What is the position of the first element from posexplode?
Show the answer
0. posexplode is 0-based.
2. Which keeps a trip whose stops array is empty?
Show the answer
posexplode_outer. The _outer variant emits one row with nulls for an empty array.
3. How do you name both output columns of posexplode?
Show the answer
.alias("pos", "stop"). Multi-column generators accept one alias per output column.
Practice it
Interview problems that use posexplode: write the PySpark, run it, and get graded on hidden tests.
Go deeper
Primary sources: functions.posexplode · functions.posexplode_outer