Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Partitioning & Pruning

repartition() vs partitionBy()

Mediumrepartitionpartitionbypartitions

Interview question

What is the difference between repartition() and partitionBy() in PySpark?

How to answer

The short answer

They both involve "," but at different layers.

repartition(...) (DataFrame API)

Controls how many in-memory partitions (and optionally by which keys) Spark uses for execution of the next stages. It triggers a when redistributing.

partitionBy(...) (writer API)

Controls directory layout on disk when writing a table/files (Hive-style partitions).

Python
# Execution partitioning (shuffle now)
df2 = df.repartition(100, "user_id")

# Storage partitioning (on write)
df2.write.mode("overwrite").partitionBy("event_date").parquet("/lake/events")

Diagram

Output
repartition(user_id)     -> Spark partitions for compute
partitionBy(event_date)  -> /event_date=2024-01-01/files...

Interview tip

You often to control file count / balance, and partitionBy so readers can prune by date/region. Do not partition storage by an ultra-high key.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill