Skip already-committed messages after a replay

Python data engineering interview problem. Difficulty: intermediate. Pattern: Deduplication. About 12 minutes. Part of the Pro drill bank.

Production ticket: A consumer restarted and re-read from the broker. Offset order isn't delivery order - sort before you decide what's new. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

A streaming consumer crashed and restarted. It reads the last batch again, and the messages can arrive out of order. The consumer already committed up to a certain offset, and processing those messages again would create duplicates downstream. messages is a list of (offset, body) pairs. Keep only the messages with an offset strictly greater than last_committed_offset, sort them by offset from small to large, and return their bodies. Write unprocessed(messages: list[tuple[int, str]], last_committed_offset: int) -> list[str]. For [(3, 'c'), (1, 'a'), (2, 'b'), (4, 'd')] with a committed offset of 2, offsets 1 and 2 are skipped, and the answer is ['c', 'd']. The starter file already has the function and the print lines at the bottom. Keep them, because the grader reads what they print.

Requirements

• Drop messages with offsets less than or equal to the committed offset.
• Sort the rest by offset ascending.
• Return only the bodies.
• Return [] when nothing is left.

Constraints

• Offsets are whole numbers.
• The batch may arrive in any order.

Examples

Input: unprocessed([(3, "c"), (1, "a"), (2, "b"), (4, "d")], 2) Output: ['c', 'd'] Offsets 1 and 2 were already committed; 3 and 4 are new, returned in offset order despite arriving out of order.

Topics: streaming, kafka, offsets, idempotency.

More Python interview questions · All interview problems · Learn data engineering