Running Total

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Window Functions. About 16 minutes. Part of the Pro drill bank.

Running sum of amount per user ordered by ts. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

A running total adds up the values so far. For example, for amounts 10, 40 and 40, the running totals are 10, 50 and 90. Each user has their own running total, in time order. For each user_id, calculate running_total, the sum of amount ordered by ts. Return user_id, ts, amount and running_total, sorted by user_id and ts. The editor creates the table df and ends with result = df. Change the last line so result has the new column. Use a window with partitionBy and orderBy, and apply F.sum over it.

Requirements

• Use a window partitioned by user_id and ordered by ts.
• Add running_total with F.sum("amount").over(window).
• Return user_id, ts, amount, running_total.
• Sort by user_id, ts and store the DataFrame in result.

Constraints

• The window must have an orderBy. Without it the sum covers the whole group.
• Each user is calculated on their own, so the totals restart at 0 for every user.
• amount is a decimal number.

Examples

Input: events ordered by ts Output: user_id | ts | amount | running_total 1 | 2024-01-01 09:00:00 | 10.0 | 10.0 1 | 2024-01-01 10:00:00 | 40.0 | 50.0 1 | 2024-01-01 11:00:00 | 40.0 | 90.0 Default sum frame accumulates through the current row.

Topics: lakebench, pyspark, sum, window.

More PySpark interview questions · All interview problems · Learn data engineering