System design interview

Handle Data Skew in Spark Join

MediumLocked55 min read

One user_id has 90% of transactions - fix Spark join skew with salting, broadcast joins, and AQE.

system-designsparkskewperformance

Start here

The problem in plain words

Imagine a supermarket with 200 checkout lanes and 100 customers. If the customers spread out evenly, everyone is done quickly. Now suppose 90 of them all join one lane. One cashier works nonstop while the other 199 stand idle, and the store cannot close until that one lane is empty.

Spark has the same problem when it joins two big tables on a key such as user_id. Spark sends all rows with the same key to the same worker task. If one user, or a null value, owns 90% of the rows, that one task is flooded while the rest finish immediately. The whole job waits for the slowest task.

This is called data skew. Adding more machines does not help, because the extra machines have nothing to do. The fix is to change how the work is divided.

Words you will meet

Spark
A system that processes large data by splitting it into pieces and working on them in parallel on many machines.
Partition and task
A partition is one piece of the data. A task is a unit of work on one partition.
Shuffle
Moving rows between machines so that rows with the same key end up together, as a join needs.
Skew
Data spread unevenly across partitions, with a few keys holding most rows.
Broadcast join
Sending a small table to every worker so the large table does not have to be shuffled.
Salting
Adding a random suffix to a hot key so its rows spread across many partitions.
AQE
Adaptive query execution: Spark adjusts the plan while running, and can split oversized join partitions.
Null key
Rows with no key value. They never match in a join, but they all land in one partition.
Spill
When a task runs out of memory and writes data to disk, making it far slower.
One lane gets 90% of the customers
U1othersslow100M rowsjoin on user_idTask 190M rows, user U1Tasks 2 to 20010M rows total, idleJob finisheswhen Task 1 is done
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.Rows are sent to tasks by user_id. User U1 owns 90 million of the 100 million rows, so one task gets nearly everything.
  2. 2.The other 199 tasks share the remaining 10 million rows and finish in moments, then sit idle.
  3. 3.The whole job has to wait for the single overloaded task. Adding machines would not change that.

The numbers, as things you can picture

  • 90 million of 100 million rows on one keyA single hot user, or a null value, holds nearly the entire table.

What the interviewer wants

Diagnose skew in a Spark join and describe the ways to fix it, including how you would decide between them.

Interview room

One user_id has 90% of transactions. Handle Spark join skew with salting, broadcast, AQE. Diagnose first.

Basics

Skew = uneven partition work. UI straggler signature. Joins co-locate keys. Broadcast small side. Salt hot keys. AQE may split skew partitions.

Prompt

Explain stragglers; salt; broadcast; AQE; measure; celebrity vs null.

Good

UI story; histogram; choose strategy; verify.

Clarify/scope

Other side size; null vs celebrity; Spark AQE version.

Scoring lens: Handle Data Skew in Spark Join

Interviewers reward incident-first teaching, a clear control loop, and named owners. Tool logos without that loop score poorly.

Weak vs strong answers on Handle Data Skew in Spark Join

Weak: list products. Strong: tell the failure story, define the mechanism, draw the path, state rollout and monitors.

Scope control

Say what is out of scope for v1. Come back only if follow-ups demand it. This is a seniority signal for Handle Data Skew in Spark Join.

Clarifying then committing

Ask a few high-use questions about SLAs, owners, and scale, then sketch. Endless questions without a diagram look evasive.

Verification checklist

After mitigation: task duration histogram flattens, max task time drops, spill does not explode, output counts match a sampled pre-fix baseline. Without verification, salting can silently corrupt joins.

Hot-key isolation pattern

Filter the top K keys into a dedicated path with more parallelism or a different join strategy, union with the normal path. Useful when a tiny key set dominates.

Null bucket specialty

If nulls are 40% of keys, salting nulls or separating null processing is mandatory. Treating null as one key is a classic interview miss.

Interview framing detail 1

For lb-sd-31, spend the first minutes making the problem concrete: who gets hurt when this fails, what SLA is implied, and what is explicitly out of scope. Then teach the prerequisite concept before proposing boxes. A candidate who names vendors first usually loses the plot. Write two clarifying questions on the board and answer them with assumptions if the interviewer shrugs. Keep the first diagram small enough to redraw when a follow-up changes a constraint.