Write Stream to Delta Lake

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Medallion. About 20 minutes. Part of the Pro drill bank.

In a medallion pipeline, the bronze layer is an append-only copy of the raw data. New batches are added to the end, and old rows are never changed. In real streaming code you would write to a Delta table with a checkpoint folder, so that if the job restarts, it continues where it stopped.

The table bronze holds the events already stored, and batch holds a new batch of events. Append the new batch to the existing data with union, as a stand-in for an append-only write. Print a checkpoint path as a text line, for example checkpoint: /tmp/lakebench/checkpoints/click_events, to show where a real job would remember its progress. Return the combined table, sorted by event_id. The editor creates both tables and ends with result = bronze. Change the last line to build your answer.

Requirements

• Combine bronze and batch with union.
• Print a checkpoint path as a text line.
• Sort the combined table by event_id.
• Store the DataFrame in result.

Constraints

• Both tables have the same columns: event_id and event_type.
• Append-only means old rows stay. The result has all 4 rows.
• union keeps duplicates. It does not remove repeated rows.

Examples

Input: bronze + batch Output: event_id | event_type 1 | click 2 | view 3 | purchase 4 | click Append semantics without a Delta writer.

Topics: lakebench, pyspark, union, checkpoint.

More PySpark interview questions · All interview problems · Learn data engineering