Load only the new rows

Python data engineering interview problem. Difficulty: beginner. Pattern: Arrays. About 10 minutes. Free to practice.

Pick the rows that arrived since the last run and report the new watermark to remember. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

An incremental pipeline loads only the rows it has not seen before. It remembers the largest id it loaded last time. Implement new_rows(rows, last_id). rows is a list of dicts, each with an integer id. Keep the rows whose id is greater than last_id, ordered by id. Return a tuple (new_rows, new_last_id). new_last_id is the largest id among the kept rows. If no row is new, it stays last_id.

Requirements

  • Only rows with id greater than last_id are kept.
  • The new rows are ordered by id.

Constraints

  • Ids are unique integers.
  • The input list is not changed.

Examples

Input: new_rows([{'id': 1}, {'id': 2}], 2) Output: ([], 2) Nothing is above 2, so there are no new rows and the watermark stays 2.

Topics: lakebench, python, incremental, watermark, pipeline.

More Python interview questions · All interview problems · Learn data engineering

beginner

Load only the new rows

Interview-style drill: Pick the rows that arrived since the last run and report the new watermark to remember.

An incremental pipeline loads only the rows it has not seen before. It remembers the largest `id` it loaded last time. Implement `new_rows(rows, last_id)`. - `rows` is a list of dicts, each with an integer `id`. - Keep the rows whose `id` is greater than `last_id`, ordered by `id`. - Return a tuple `(new_rows, new_last_id)`. - `new_last_id` is the largest `id` among the kept rows. If no row is new, it stays `last_id`.