Share of total revenue without collecting the total
PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Driver Memory. About 15 minutes. Part of the Pro drill bank.
Compute each customer's share of overall revenue with the total staying inside Spark. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.
df is the orders table: order_id, customer_id and amount. In production it has billions of rows. For each customer return their total revenue and share, their revenue divided by the revenue of all customers, rounded to 2 decimals. Sort by revenue descending, then customer_id. Assign the DataFrame to result.
Requirements
- Return the columns customer_id, revenue, share.
- Sort by revenue descending, then customer_id.
Constraints
- df stands for a table with billions of rows.
- Only the final, small result may be brought to the driver.
- amount is positive.
Examples
Input: df order_id | customer_id | amount 1 | A | 50 2 | A | 30 3 | B | 15 4 | C | 5 Output: customer_id | revenue | share A | 80 | 0.8 B | 15 | 0.15 C | 5 | 0.05 Total revenue is 100. A has 80 (0.8), B has 15 (0.15) and C has 5 (0.05).
Topics: lakebench, pyspark, driver, crossJoin, share of total.
More PySpark interview questions · All interview problems · Learn data engineering
Share of total revenue without collecting the total
Interview-style drill: Compute each customer's share of overall revenue with the total staying inside Spark.
`df` is the orders table: `order_id`, `customer_id` and `amount`. In production it has billions of rows. For each customer return their total `revenue` and `share`, their revenue divided by the revenue of all customers, rounded to 2 decimals. Sort by `revenue` descending, then `customer_id`. Assign the DataFrame to `result`.