When to Cache

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Explain Plans. About 20 minutes. Part of the Pro drill bank.

Cache a frame reused by two aggregations and union the summaries. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Cache is useful when you use the same DataFrame for several different calculations. Here you will compute two summaries from the same table, so it is worth keeping the table in memory once. Cache df. Compute the total amount for each region, and the total amount for each date. Put the two results into one table with a grain column that says which kind of total each row is: region or date. For the date rows, put the date text in the region column, so both parts have the same columns: grain, region and total. Sort by grain, then region. The editor creates the table df and ends with result = df. Change the last line to build your answer. Use union to stack the two results.

Requirements

• Cache df once and reuse the cached DataFrame for both summaries.
• Make one summary by region and one by date, each with a total.
• Add the grain column (region or date) and put dates in the region column for the date rows.
• Stack them with union, select grain, region, total, sort by grain, region, and store it in result.

Constraints

• Both parts of a union must have the same columns in the same order.
• The result has 3 region rows and 3 date rows, 6 rows in total.
• union matches columns by position, not by name.

Examples

Input: reuse df Output: grain | region | total date | 2024-01-01 | 30.0 date | 2024-01-02 | 70.0 date | 2024-01-03 | 110.0 region | APAC | 60.0 region | EU | 60.0 region | US | 90.0 Two actions share a cached base.

Topics: lakebench, pyspark, cache, reuse.

More PySpark interview questions · All interview problems · Learn data engineering