Cache vs Persist

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Explain Plans. About 20 minutes. Part of the Pro drill bank.

Cache df then aggregate totals by region. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Spark does not calculate a DataFrame until it needs the answer, and if you use the same DataFrame twice, it calculates it twice. cache() tells Spark to keep the data in memory after the first time, so the second use is fast. persist() is the same idea, but lets you choose where to keep it (memory, disk, or both). Call cache() on df, then group by region and add up amount as total. Sort by region. The simulator prints a short note when you cache. The editor creates the table df and ends with result = df. Change the last line to build your answer.

Requirements

• Call df.cache() and use the cached DataFrame.
• Group by region and sum amount as total.
• Sort by region.
• Store the DataFrame in result.

Constraints

• cache() does not change the data. It only changes how Spark stores it.
• The table has 6 rows and 3 regions: US, EU and APAC.
• The result has 1 row for each region.

Examples

Input: regional amounts Output: region | total APAC | 60.0 EU | 60.0 US | 90.0 Caching does not change aggregate math.

Topics: lakebench, pyspark, cache.

More PySpark interview questions · All interview problems · Learn data engineering