Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. What is idempotency?

Airflow & DAGs · Core Concepts & DAGs

What is idempotency?

Mediumairflow-08
idempotencyretries

Question

What is idempotency, and why does it matter in Airflow pipelines?

Solution

Idempotency means:

> Running the same operation multiple times produces the same final result.

If a load of 1000 records fails halfway and Airflow retries with a blind INSERT, you may end up with 2000 rows.

Safer patterns:

MERGE / UPSERT
DELETE + INSERT for a partition
Replace partition
Unique / natural keys

Interview tip: Airflow retries are only safe if tasks are idempotent (or compensated).

PreviousNext