Solve “Explode Array Column”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

How do you explode an array?

Mediumexplodearraysdataframe-api

Interview question

How do you explode an array column in PySpark?

How to answer

The short answer

`explode` turns each array element into its own row (and optionally structs via explode on array<struct>). Use explode_outer to keep rows with null/empty arrays.

Example

Python
from pyspark.sql import functions as F

df = spark.createDataFrame(
    [(1, ["a", "b"]), (2, ["c"]), (3, None)],
    ["id", "tags"],
)

# Drops id=3 (null array) by default
df.select("id", F.explode("tags").alias("tag")).show()

# Keeps id=3 with tag=null
df.select("id", F.explode_outer("tags").alias("tag")).show()

# posexplode gives position
df.select("id", F.posexplode("tags").alias("pos", "tag")).show()

Diagram

Output
id | tags
1  | [a, b]     explode =>   id | tag
2  | [c]                     1  | a
                             1  | b
                             2  | c

Nested / map variants

  • explode on map yields key/value (use explode carefully; prefer explode_outer).
  • For arrays, parse with from_json then explode.

Interview tip

Mention blow-up: exploding large arrays multiplies rows and can cause skew or OOM if unchecked.

Null and empty arrays, with real output

The input above has a null array, and real data also has empty ones. Using 4 rows, (1, [a, b]), (2, [c]), (3, null) and (4, []), on PySpark 3.5.3:

Output
explode        ->  1 a | 1 b | 2 c                       (ids 3 and 4 are gone)
explode_outer  ->  1 a | 1 b | 2 c | 3 NULL | 4 NULL
rows in: 4 | explode rows: 3 | explode_outer rows: 5

Both ids vanish under plain explode, with no warning. If you later join back to the source or count customers, they silently go missing. Use explode_outer whenever a row with no elements must still appear, and posexplode when the position inside the array matters.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill