Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

How do you pivot a DataFrame?

Mediumpivotreshapeaggregation

Interview question

How do you pivot a DataFrame in PySpark?

How to answer

The short answer

Pivot turns unique values of a column into new columns, usually with an aggregation. Classic use: reshape event counts or monthly metrics into a wide report.

Example

Python
from pyspark.sql import functions as F

sales = spark.createDataFrame(
    [
        ("east", "2024-01", 10),
        ("east", "2024-02", 12),
        ("west", "2024-01", 8),
    ],
    ["region", "month", "amount"],
)

wide = (
    sales.groupBy("region")
    .pivot("month", ["2024-01", "2024-02"])  # optional explicit values
    .sum("amount")
)
wide.show()

Diagram

Output
region month   amount          region  2024-01  2024-02
east   2024-01 10       =>     east    10       12
east   2024-02 12              west    8        null
west   2024-01 8

Tips

  • Pass explicit pivot values when known; discovering distinct values requires an extra pass.
  • Pivot implies aggregation (sum, count, agg).
  • Inverse is stack / melt-style unpivot with expr("stack(...)") or pandas-style reshaping outside Spark.

Interview tip

Wide pivots with high explode column count and can hurt planning; keep pivot keys bounded.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill