Conditional Aggregation

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Aggregation. About 16 minutes. Part of the Pro drill bank.

Electronics and Clothing revenue in one aggregate row. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Conditional aggregation means adding up a value only for rows that meet a condition. It lets you build several totals in one pass over the data, one for each condition. From df, return 1 row with 2 columns: electronics_revenue, the sum of amount for rows where product_category is Electronics, and clothing_revenue, the sum of amount for rows where it is Clothing. Other categories, like Food, are not counted in either. The editor creates the table df and ends with result = df. Change the last line to build your answer. A useful tool is F.when(condition, value).otherwise(0) inside F.sum.

Requirements

• Return exactly 1 row with the columns electronics_revenue and clothing_revenue.
• Add amount into electronics_revenue only for the Electronics category.
• Add amount into clothing_revenue only for the Clothing category.
• Ignore all other categories, and store the DataFrame in result.

Constraints

• The category names are Electronics and Clothing, with a capital first letter.
• Use F.sum(F.when(...).otherwise(0)) or add a helper column first.
• The result has 1 row, so use agg without a groupBy or group by a constant.

Examples

Input: category amounts Output: electronics_revenue | clothing_revenue 220.0 | 70.0 Food is ignored by both conditional sums.

Topics: lakebench, pyspark, when, sum.

More PySpark interview questions · All interview problems · Learn data engineering