Small Files Problem

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Medallion. About 20 minutes. Part of the Pro drill bank.

The small files problem happens when a job writes hundreds of tiny files instead of a few big ones. Every file costs time to open, so queries become slow. The number of output files is usually the number of partitions, so fewer partitions mean fewer files.

The table df has 6 rows. Reduce it to a single partition with coalesce(1), which is the stand-in here for writing a single output file. Then select id, date, region and amount, and sort by id. The data stays the same. Only the number of partitions changes. The editor creates the table df and ends with result = df. Change the last line to build your answer.

Requirements

• Call df.coalesce(1).
• Select id, date, region, amount.
• Sort by id and store the DataFrame in result.

Constraints

• All 6 rows and all 4 columns stay.
• coalesce(1) puts all rows in 1 partition, which means one output file.
• Use coalesce, not repartition(1), to avoid a full shuffle.

Examples

Input: 6 rows Output: id | date | region | amount 1 | 2024-01-01 | US | 10.0 2 | 2024-01-01 | EU | 20.0 3 | 2024-01-02 | US | 30.0 4 | 2024-01-02 | EU | 40.0 5 | 2024-01-03 | US | 50.0 6 | 2024-01-03 | APAC | 60.0 Rows unchanged after coalesce(1).

Topics: lakebench, pyspark, coalesce.

More PySpark interview questions · All interview problems · Learn data engineering