Intermediate · 30 min · 5 steps
Spreading one huge customer across many workers so a job stops getting stuck
Vikram Sen, Principal Big Data Engineer at OmniRetail India
It is 11:30 PM on the first night of the Great Indian Festive Sale. Vikram Sen, a senior big data engineer at OmniRetail India, gets an automatic alert on his phone. OmniRetail is an online marketplace that handles tens of millions of checkout events during festival sales.
The question
How can we split one very busy join key into smaller pieces so that every worker in the cluster gets a fair share of the work?
- Data skew
- Key salting
- Spark shuffle
- Distributed joins
The rest of this case study is locked
The data, every step with its code and result, and the final answer open with a pass. A few case studies stay free so you can see how they work.