Repartition vs Coalesce

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Explain Plans. About 20 minutes. Part of the Pro drill bank.

Shrink partitions with coalesce(2) and return the ordered frame. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

A Spark DataFrame is split into partitions, and each partition is processed by one task. After a filter, you may be left with many small partitions, which is wasteful. repartition(n) can change the number of partitions up or down, but it always does a full shuffle (it moves data around the network). coalesce(n) only reduces the number of partitions, and it avoids a full shuffle by merging neighbours. When you only want fewer partitions, coalesce is cheaper. Call coalesce(2) on df, then select id, date, region and amount, and sort by id. The data does not change, only how it is split. The editor creates the table df and ends with result = df. Change the last line to build your answer. The simulator prints a note when you call coalesce.

Requirements

• Reduce the partitions with df.coalesce(2).
• Select id, date, region, amount.
• Sort by id and store the DataFrame in result.

Constraints

• coalesce can only lower the number of partitions. It cannot raise it.
• The rows and values must stay exactly the same, with 6 rows.
• Use coalesce, not repartition, because you are only reducing.

Examples

Input: partitioned-ish fixture Output: id | date | region | amount 1 | 2024-01-01 | US | 10.0 2 | 2024-01-01 | EU | 20.0 3 | 2024-01-02 | US | 30.0 4 | 2024-01-02 | EU | 40.0 5 | 2024-01-03 | US | 50.0 6 | 2024-01-03 | APAC | 60.0 Data unchanged; coalesce only affects partition count.

Topics: lakebench, pyspark, coalesce, repartition.

More PySpark interview questions · All interview problems · Learn data engineering