explode
Turn each array element into its own row, and the rows explode silently drops.
On this page
Show code in
Every code block on the page follows this.
You will learn
- How explode turns array elements into rows
- Why rows with empty or null arrays disappear, and how explode_outer keeps them
- How to explode maps and split strings first
- Why explode can blow up data size
F.explode(arr) outputs one row per array element, copying the other columns. Rows whose array is empty or null produce no rows at all; explode_outer keeps them with a null.What it does
explode is a generator: one input row can produce zero, one or many output rows. Each output row holds one element of the array, alongside copies of the other selected columns.
Step by step
orders
| order_id | items |
|---|---|
| 1 | [pen, ink] |
| 2 | [paper] |
| 3 | [] |
| 4 | null |
explode(items)
| order_id | item |
|---|---|
| 1 | pen |
| 1 | ink |
| 2 | paper |
Orders 3 and 4 vanish: an empty array has no elements to emit, and neither does null.
Run the example
from pyspark.sql import functions as F result = orders.select("order_id", F.explode_outer("items").alias("item"))
SELECT order_id, item FROM orders LATERAL VIEW OUTER explode(items) t AS item
Switch to PySpark to edit and run this example in your browser.
This runs explode_outer, so orders 3 and 4 stay with a null item. Change it to F.explode to see them disappear.
The explode family
| Function | Output | Empty / null array |
|---|---|---|
explode | One row per element | Row dropped |
explode_outer | One row per element | One row with null |
posexplode | Position and element | Row dropped |
posexplode_outer | Position and element | One row with nulls |
inline | One row per struct, struct fields as columns | Row dropped |
On a map column, explode produces two columns, key and value.
Strings first: split
Comma-separated text is a string, not an array. F.split("tags", ",") turns it into one; trim the elements if the separator has spaces: F.explode(F.split("tags", r"\s*,\s*")). split takes a regular expression, so a separator like | must be escaped: "\\|".
In SQL
Spark SQL accepts explode directly in the SELECT list (SELECT order_id, explode(items) AS item) and the Hive-style LATERAL VIEW shown above. LATERAL VIEW OUTER is the explode_outer equivalent. Only one generator is allowed per SELECT list.
Under the hood: data size
explode is a narrow transformation: no shuffle. But it multiplies rows. Exploding a column averaging 1,000 elements turns 1 million rows into 1 billion, each carrying a copy of every other column. Select only the columns you need before exploding, and if you will aggregate straight back, check whether a higher-order function such as aggregate, filter or transform can work on the array without exploding at all.
Common mistakes
Losing rows with empty arrays
Exploding two arrays in one select
arrays_zip.Splitting on a regex metacharacter
split(col, "|") splits on every character, because | is regex alternation. Escape it.Key takeaways
- explode makes one row per array element and copies the other columns.
- Empty and null arrays produce no rows; explode_outer keeps them.
- posexplode also returns each element's position.
- explode does not shuffle, but it multiplies rows: prune columns first.
Check yourself
3 questions1. A table has 3 rows with arrays of size 2, 0 and null. How many rows does explode return?
Show the answer
2. Only the first row has elements. The empty and null arrays produce nothing. explode_outer would return 4.
2. Which keeps a row whose array is empty?
Show the answer
explode_outer. The _outer variants emit one row with nulls for an empty or null array.
3. Does explode cause a shuffle?
Show the answer
No, it is a narrow transformation, but it multiplies rows. Each input row is expanded in place. The cost is the extra data, not network movement.
Practice it
Interview problems that use explode: write the PySpark, run it, and get graded on hidden tests.
Go deeper
posexplodePySpark functions
collect_list / collect_setSpark internals
Narrow vs wide transformations
Primary sources: functions.explode · LATERAL VIEW