PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Window Functions. About 16 minutes. Part of the Pro drill bank.
Days since previous event per user via lag. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.
For each user_id, use lag(ts) and datediff to compute time_since_last_event (days). Return user_id, ts, time_since_last_event. First events are null. Order by user_id, ts. Assign result.
Input: timestamps per user Output: user_id | ts | time_since_last_event 1 | 2024-01-01 10:00:00 | None 1 | 2024-01-01 12:00:00 | 0 1 | 2024-01-01 15:00:00 | 0 2 | 2024-01-02 09:00:00 | None 2 | 2024-01-02 10:00:00 | 0 Same calendar day yields datediff 0; first rows stay null.
Topics: lakebench, pyspark, lag, datediff.
More PySpark interview questions · All interview problems · Learn data engineering
Interview-style drill: Days since previous event per user via lag.
For each `user_id`, use `lag(ts)` and `datediff` to compute `time_since_last_event` (days). Return `user_id`, `ts`, `time_since_last_event`. First events are null. Order by `user_id`, `ts`. Assign `result`.