Partition by Column

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Medallion. About 20 minutes. Part of the Pro drill bank.

When you write a big table to storage, you often partition it by a column such as the date, so a query for 1 day only reads that day's folder. This simulator has no Parquet writer, so you will practise the first half of the idea: repartition the data by the same columns you would use for the folders.

The table df has id, date, region and amount. Repartition it into 3 partitions using the columns date and region with repartition(3, "date", "region"). This puts rows with the same date and region in the same partition, like partition folders on disk. Then select id, date, region and amount, sorted by id. The rows themselves do not change. The editor creates the table df and ends with result = df. Change the last line to build your answer.

Requirements

• Call repartition(3, "date", "region") on df.
• Select id, date, region, amount.
• Sort by id and store the DataFrame in result.

Constraints

• Repartition by the 2 columns together, date and region.
• All 6 rows and all 4 columns must stay in the result.
• There is no file writing in this simulator, so only the DataFrame is checked.

Examples

Input: sales by date/region Output: id | date | region | amount 1 | 2024-01-01 | US | 10.0 2 | 2024-01-01 | EU | 20.0 3 | 2024-01-02 | US | 30.0 4 | 2024-01-02 | EU | 40.0 5 | 2024-01-03 | US | 50.0 6 | 2024-01-03 | APAC | 60.0 Shim prints a repartition note; row set stays intact.

Topics: lakebench, pyspark, repartition.

More PySpark interview questions · All interview problems · Learn data engineering