Identify Skew

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Skew. About 20 minutes. Part of the Pro drill bank.

Data skew means some keys have far more rows than others. In Spark, all rows of one key are handled by one task, so one very busy key can make a job slow while the other tasks wait. The first step to fix skew is to find the hot keys, by counting rows per key.

The table df has user_id and item. Count how many rows each user has as row_count. Sort by row_count from high to low, and then by user_id, so the busiest users come first and the order is stable. The hot key (the user with far more rows than the others) should be easy to see at the top. The editor creates the table df and ends with result = df. Change the last line to build your answer.

Requirements

• Group by user_id and count the rows as row_count.
• Sort by row_count descending, then user_id.
• Store the DataFrame in result.

Constraints

• Count rows with F.count("*").
• The sample has 6 rows and 3 users.
• The second sort key (user_id) makes the order stable when counts are equal.

Examples

Input: skewed keys Output: user_id | row_count 1 | 4 2 | 1 3 | 1 user_id 1 dominates the fixture.

Topics: lakebench, pyspark, skew, counts.

More PySpark interview questions · All interview problems · Learn data engineering