PySpark production problems: data skew explained with visuals

This guide teaches data skew the way you meet it at work: through a real scenario, pictures of what goes wrong, small pieces of code and a production-style fix. If you are new to Spark, read what is Apache Spark first for the words partition, task and executor.

What you will learn

• Why Spark jobs sometimes get stuck at 99%.
• What data skew means in simple terms.
• How partitions, shuffles and joins contribute to the problem.
• How to detect skew using the Spark UI.
• How to solve skew using salting, Adaptive Query Execution (AQE) and other techniques.
• How to explain the solution in a data engineer interview.

1The real-world problem: a job that never finishes

Imagine you are a data engineer at an e-commerce company. Your manager asks you to generate a report showing the total orders received from each country. Your dataset contains 10 million orders. Here is a small piece of it:

You want the number of orders from each country.

orders.groupBy("country").count().show()

The output might look like this:

These counts are illustrative, to show the shape of the problem. The code looks simple. But the dataset is much larger than the example, and Spark must process millions of records.

What you expect

You expect Spark to distribute the work among multiple workers so that everything finishes quickly.

What actually happens

Imagine the Spark UI shows this:

Most tasks finished in 2 minutes. One task is still running after 30 minutes. Why? Because Spark can distribute work across workers, but the work is not always distributed evenly. That brings us to the main concept.

2What is data skew?

Data skew occurs when some Spark partitions contain significantly more data than others. Think about four checkout counters at a supermarket.

Spark experiences the same problem. A partition with millions of records can take much longer than a partition with a few thousand records.

Why does this happen?

Common causes include:

• One key appears much more frequently than other keys.
• Many records have a NULL key.
• A few customers or countries dominate the dataset.
• A join key has a highly uneven distribution.

3How Spark processes this data

Before fixing data skew, understand three things: partitions, shuffles and tasks.

Partitions

A partition is a chunk of data Spark can process independently.

Shuffles

Now consider this operation:

orders.groupBy("country").count()

Spark must bring records with the same country key together to calculate the correct count. For example, records with country = "US" must contribute to the same final US count. This redistribution of data between partitions is called a shuffle.

If one key dominates the data, a shuffle can create an uneven workload. For groupBy, Spark must ultimately combine the counts for each key. For joins, matching keys must be brought together according to the join strategy.

4How do you identify data skew?

Do not assume every slow Spark job is caused by data skew. First, inspect the Spark UI. Open the Stages tab and focus on these metrics:

For example, suppose a stage has 200 tasks:

This is a strong sign of possible skew, especially if the slow task also has an unusually high shuffle read size or number of records processed. However, a slow task can also result from garbage collection, disk I/O, network delays, executor contention or other issues. Check the task metrics before choosing a fix.

5Solution one: salting for a skewed aggregation

Let's solve the original groupBy problem. Suppose 90% of your orders belong to the US. Rather than performing one large aggregation for that key in a single final group, we can divide its work into smaller groups.

Step 1: Understand the idea

The salt values make the grouping keys different: (US, 0), (US, 1), (US, 2) and (US, 3). This allows Spark to distribute the intermediate aggregation across more partitions.

Step 2: Implement salting in PySpark

Here is a simplified example for counting orders, with four salt buckets.

from pyspark.sql import functions as F

N = 4

# The salt comes from the order id, so the same row always gets the same salt
salted_orders = orders.withColumn(
    "salt",
    F.pmod(F.hash("order_id"), F.lit(N))
)

# First aggregation: split the work across salt buckets
partial_counts = (
    salted_orders
    .groupBy("country", "salt")
    .count()
)

# Second aggregation: combine the partial counts
final_counts = (
    partial_counts
    .groupBy("country")
    .agg(F.sum("count").alias("total_orders"))
)

final_counts.show()

Let's understand it line by line.

• F.hash("order_id") turns each order id into a number, and F.pmod(..., N) keeps it between 0 and 3, giving the salt.
• groupBy("country", "salt") calculates intermediate counts for separate buckets.
• groupBy("country") combines those counts into the final country totals.

The output still contains one final count per country. Salting changes how the work is distributed, not the intended result.

A real run of this idea, with printed partition sizes, is in the scenario guide: on a small test the largest partition dropped from 1,010 rows to 477.

6Solution two: salting for a skewed join

A skewed join is another common production problem. Imagine two datasets. The orders table has 10 million records, with millions of orders from the US. The country lookup table has only a few rows, one per country.

orders.join(country_lookup, on="country", how="left")

If the join strategy sends a huge number of records for a hot key through the same join partition, one task may become a straggler.

The join-salting strategy

For a sufficiently large skewed dataset joined to a smaller lookup table, the general technique is:

1. 1Add a salt value to rows in the large table.
2. 2Replicate lookup rows across all possible salt values.
3. 3Join using both the original key and the salt.

Example implementation

from pyspark.sql import functions as F

N = 4

# Add a salt to each order
salted_orders = orders.withColumn(
    "salt",
    F.pmod(F.hash("order_id"), F.lit(N))
)

# Replicate each lookup row across salt values 0 to N-1
salts = spark.range(N).select(
    F.col("id").cast("int").alias("salt")
)

salted_lookup = country_lookup.crossJoin(salts)

# Join on both country and salt
result = salted_orders.join(
    salted_lookup,
    on=["country", "salt"],
    how="left"
).drop("salt")

This is a conceptual implementation of join salting. It is most appropriate when the lookup table is small enough to replicate, and the salt distribution suits the skewed keys. A few important details:

• Every large-table row must receive a valid salt value.
• The lookup table must contain a matching copy for every salt value used.
• If only a few keys are skewed, salting only those keys reduces unnecessary replication.
• A normal broadcast join may be simpler and faster when the lookup table fits safely in the available memory.
• For a many-to-many join, salting does not remove the output-size problem. If one key matches millions of rows on both sides, the join can legitimately produce an enormous result.

7Solution three: let Spark handle skew with AQE

You do not always need to implement salting by hand. Spark has a feature called Adaptive Query Execution (AQE). It uses runtime statistics to adjust parts of the execution plan while a query is running. For supported skewed sort-merge joins, AQE can split skewed partitions into smaller pieces and process them more efficiently.

You can enable AQE and skew-join optimization like this:

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

These settings are commonly enabled by default in modern Spark versions, but verify the configuration for your environment.

How AQE helps

8Other ways to solve data skew

Salting is useful, but it should not be your first solution for every workload.

One thing to remember: increasing spark.sql.shuffle.partitions alone does not necessarily fix a hot key. All records for a given key can still be assigned to the same hash partition during a normal hash-based shuffle.

9How to debug a real production issue

Imagine your daily pipeline normally finishes in 10 minutes, but today it takes 90 minutes. Use this checklist:

1. 1Find the slow stage. Open the Spark UI and identify which stage dominates the execution time.
2. 2Compare task metrics. Compare task duration, shuffle read size, records processed and spill metrics.
3. 3Inspect the execution plan. Use explain("formatted") and check for exchanges, join strategies and aggregation stages.
4. 4Inspect the key distribution. Check whether one customer, country or null key accounts for a disproportionate share of the records.
5. 5Try the simplest appropriate fix. Evaluate AQE, a safe broadcast join, pre-aggregation or selective salting.
6. 6Validate correctness and performance. Compare output counts and join results, then benchmark on representative data.

You can also inspect the physical plan directly:

orders.groupBy("country").count().explain("formatted")

The plan helps you understand what Spark intends to do. The Spark UI tells you what happened at runtime. Use both rather than diagnosing skew from the code alone.

10Interview questions and answers

Q1. What is data skew in Spark?

Data skew occurs when data is unevenly distributed across partitions, causing a few tasks to take significantly longer than others.

Q2. Why does a Spark job get stuck at 99%?

Most tasks may have completed, while a small number of slow tasks are still processing. Skew is one possible cause. Resource contention and I/O problems can also cause stragglers.

Q3. What is salting?

Salting adds an extra value to a hot key to divide its workload across multiple groups. For aggregations, the partial results are combined afterward. For joins, matching salt values must be assigned on both sides.

Q4. Does increasing the number of partitions solve skew?

Not necessarily. If a hot key is always routed to one hash partition, adding more shuffle partitions will not automatically spread that key across multiple partitions.

Q5. What is the difference between AQE and salting?

AQE can adapt supported parts of a query plan using runtime statistics. Salting is a data transformation you design to split hot keys. AQE is often worth checking first, but it does not replace every skew-handling strategy.

Q6. What are the disadvantages of salting?

It adds complexity, can increase data movement or replication, and may not help if the workload is not skewed. The final aggregation can also remain a bottleneck if the salt strategy is poorly chosen.

For more practice, read the step-by-step salting answer and try the problems Identify Skew and Fix Skew with Salting.

11Quick knowledge check

Your Spark stage has 100 tasks. 99 finish quickly, but one task processes much more shuffle data and runs for 40 minutes. What should you do first?

1. 1Increase the cluster size immediately.
2. 2Inspect the slow task's metrics and investigate data skew.
3. 3Add random salt to every column.
4. 4Increase shuffle partitions without checking anything else.

Final takeaway

Data skew is not just a Spark coding problem. It is a workload-distribution problem. The most important sequence is:

If you remember just one thing for interviews, remember this: first prove where the bottleneck is, then choose the optimization. That is the difference between knowing PySpark syntax and thinking like a production data engineer.