The short answer
`explode` turns each array element into its own row (and optionally structs via explode on array<struct>). Use explode_outer to keep rows with null/empty arrays.
Example
from pyspark.sql import functions as F
df = spark.createDataFrame(
[(1, ["a", "b"]), (2, ["c"]), (3, None)],
["id", "tags"],
)
# Drops id=3 (null array) by default
df.select("id", F.explode("tags").alias("tag")).show()
# Keeps id=3 with tag=null
df.select("id", F.explode_outer("tags").alias("tag")).show()
# posexplode gives position
df.select("id", F.posexplode("tags").alias("pos", "tag")).show()Diagram
id | tags
1 | [a, b] explode => id | tag
2 | [c] 1 | a
1 | b
2 | cNested / map variants
explodeonmapyields key/value (useexplodecarefully; preferexplode_outer).- For arrays, parse with
from_jsonthen explode.
Interview tip
Mention blow-up: exploding large arrays multiplies rows and can cause skew or OOM if unchecked.
Null and empty arrays, with real output
The input above has a null array, and real data also has empty ones. Using 4 rows, (1, [a, b]), (2, [c]), (3, null) and (4, []), on PySpark 3.5.3:
explode -> 1 a | 1 b | 2 c (ids 3 and 4 are gone)
explode_outer -> 1 a | 1 b | 2 c | 3 NULL | 4 NULL
rows in: 4 | explode rows: 3 | explode_outer rows: 5Both ids vanish under plain explode, with no warning. If you later join back to the source or count customers, they silently go missing. Use explode_outer whenever a row with no elements must still appear, and posexplode when the position inside the array matters.