Pipelines & scenarios · Core Pipeline Concepts

What is data skew?

Mediumskewstragglerssaltingjoinsperformance

Interview question

What is data skew, and how do you handle it in pipelines?

How to answer

The short answer

means a few keys hold a disproportionate share of the data. Tasks on those keys become stragglers while other tasks finish and sit idle.

Output
join / groupBy key = country

  US:     ############################  (huge)
  IN:     ##########
  DE:     ##
  IS:     #

One executor stuck on US; cluster looks "busy" but progress crawls.

Where skew shows up

  • Joins and aggregations on popular keys (null, unknown, big customers)
  • Hot in Kafka
  • Uneven file sizes after bad partitioning

Mitigations

  1. 1Filter / isolate hot keys and process them separately
  2. 2: add a random suffix to hot keys, then aggregate twice
  3. 3 the small side instead of shuffling both
  4. 4Adaptive skew join (Spark ) when available
  5. 5 with a better key or more partitions

Interview tip: Define skew as uneven key distribution causing stragglers, then name salting or broadcast as fixes. Mention checking Spark stage timelines for one long task.