Batch & Streaming · Batch Processing

What is a shuffle?

Mediumshufflejoinskewperformance

Interview question

What is a shuffle in Spark (or MapReduce), and why is it expensive?

How to answer

The short answer

A redistributes data across the cluster so that records that must be processed together (same join key, same group key) land on the same /executor.

Output
Before shuffle (partition by file):     After shuffle (by user_id):
exec1: u1,u7,u2                         exec1: all u1,u4,...
exec2: u4,u1,u9          ======>        exec2: all u2,u7,...
exec3: u2,u4                            exec3: all u9,...

Why it hurts

  • Network transfer of large bytes
  • Disk spill when memory is tight
  • Sort/hash overhead
  • Sensitive to skew (one key owns 80% of rows)

DE habits

  • Filter early before shuffle
  • Broadcast small to avoid big shuffles
  • / skew hints for hot keys
  • Right-size partitions (spark.sql.shuffle.partitions)

Interview tip: Define shuffle as " by key across the network," then name join/groupBy as causes and skew as the failure mode.