Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing

PySpark for Distributed Processing

Progress0/21
x

PySpark Architecture

  • PySpark architecture: the one explanation35m
  • When do you need Spark?12m
  • What is a cluster?12m
  • Reading and writing data12m
  • SQL inside Spark12m

Mental model & DataFrames

  • PySpark architecture & mental model10m
  • DataFrame basics10m
  • Column operations & built-in functions10m

Aggregations, windows & nested data

  • Aggregations & groupings10m
  • PySpark window functions12m
  • Joins & optimization strategies10m
  • Handling complex & nested data12m
  • Partitioning, repartition & coalesce10m
  • Medallion pipeline project14m

Performance at Scale

  • Caching & persistence12m
  • Data skew detection and salting14m
  • Reading a Catalyst physical plan12m

Production Spark

  • UDFs in depth, and why to avoid them12m
  • Structured Streaming & watermarks14m
  • Delta Lake: MERGE, time travel, ACID12m
  • Capstone part 2: incremental MERGE16m
Back to track
  1. Learn
  2. PySpark for Distributed Processing
  3. Performance at Scale
  4. Data skew detection and salting

Lesson 16 of 21 · Case study

Data skew detection and salting

pysparkadvanced14 min

Overview

One country owning most rows is a straggler. Salt the hot key so the crew can share the pallet.

Module: Performance at Scale

This section walks through the idea with a short example, then the trade-offs you should mention in an interview.

In practice you start from the raw rows, apply the transform step by step, and check the shape of the result before you move on.

A common mistake is to jump straight to the final query without naming the grain or the join keys that keep the result correct.

Once the core path works, you harden it for nulls, duplicates, and late data so the pipeline stays reliable under load.

The Pro write-up covers the full explanation, worked examples, and the code you can run in the studio.

# Locked example
result = transform(frame)
print(result.head())

This lesson requires Pro

This lesson is part of Performance at Scale. Pro opens the full lesson and the exercises.

Compare Free vs Pro