Abhinav Menon

@abhinavmenon

Free member

Student / switching in DE working on AWS. Interested in CDC.

Coimbatore, Tamil NaduStudent / switching in1 questions2 answers

AWSSQLIcebergBigQuery

Streak: 0d
XP: 657
Lessons done: 3
Studio passes: 5

First Lesson5 Tickets500 XP

Share on X Share on LinkedIn

Questions

votes

answers

views

Exactly-once semantics for a file-based ETL, or is at-least-once fine?

Files land in a folder, a script picks them up, transforms, and loads. If the script crashes mid-run and retries, some rows can get double-loaded. Is chasing true exactly-once worth it here, or is idempotent at-least-once enough?

ETL / ELTetlidempotency