Daily revenue without pulling rows to the driver

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Driver Memory. About 12 minutes. Part of the Pro drill bank.

Report paid orders and revenue per day while keeping the big table on the executors. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

df is the orders table: order_id, order_date, amount and status. In production it has billions of rows, so the driver (the single machine that runs your program and has limited memory) cannot hold them, while the executors (the many worker machines that hold the data) can. For each order_date, return the number of paid orders as orders and their total amount as revenue. Days with no paid order are not listed. Sort by order_date. Assign the DataFrame to result.

Requirements

• Return the columns order_date, orders, revenue.
• Sort by order_date.

Constraints

• df stands for a table with billions of rows.
• Only the final, small result may be brought to the driver.
• status is one of paid, refunded, cancelled.

Examples

Input: df order_id | order_date | amount | status 1 | 2024-05-01 | 120 | paid 2 | 2024-05-01 | 80 | refunded 3 | 2024-05-01 | 45.5 | paid 4 | 2024-05-02 | 60 | paid 5 | 2024-05-03 | 30 | cancelled 6 | 2024-05-03 | 75 | paid Output: order_date | orders | revenue 2024-05-01 | 2 | 165.5 2024-05-02 | 1 | 60 2024-05-03 | 1 | 75 Only paid orders count: 1 May has two (165.5), 2 May one (60) and 3 May one (75).

Topics: lakebench, pyspark, driver, groupBy, aggregation.

More PySpark interview questions · All interview problems · Learn data engineering