Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

coalesce vs repartition

Mediumcoalescerepartitionpartitionsperformance

Interview question

What is the difference between coalesce and repartition in PySpark?

How to answer

The short answer

Both change count, but they move data differently.

repartition(n) / repartition(cols...)

  • Can increase or decrease partitions
  • Full for even redistribution
  • Use before large writes when you need balanced files, or to partition by key

coalesce(n)

  • Typically used to reduce partitions
  • Avoids a full shuffle by merging existing partitions (narrow dependency)
  • Can leave uneven sizes if you shrink aggressively
Python
# Evenly redistribute to 200 partitions (shuffle)
df2 = df.repartition(200)

# Partition by key (shuffle)
df3 = df.repartition(100, "user_id")

# Reduce to 10 partitions without full shuffle
df4 = df.coalesce(10)

Diagram

Output
repartition(4):  any -> shuffle -> balanced 4
coalesce(2):     P0+P1 merge, P2+P3 merge (no full reshuffle)

When to use which

  • Writing many tiny outputs after a big filter: coalesce (or ) carefully.
  • Skewed / uneven partitions needing balance: repartition.
  • Never coalesce(1) on huge data unless you truly need a single file and can afford it.

repartition by column is about in-memory partitioning for the next stage; partitionBy on write is about directory layout on disk.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill