Intermediate · 25 min · 5 steps
Loading only the rows that changed since the last run
Ananya Hegde, Lead ETL Pipeline Engineer at OmniRetail Lakehouse
It is Wednesday, 3:00 PM. Ananya Hegde builds data loading pipelines at OmniRetail Lakehouse. OmniRetail is a big retail chain in India, with 220 superstores and a grocery app used by over 1.2 million shoppers. She has been called into a meeting about cloud costs with the director of cloud engineering. The Snowflake and AWS bill is ₹14,50,000 over the forecast.
The question
How do we read a saved 'last loaded' time, fetch only the rows that changed, handle a customer who changed many times, and move the saved time forward safely?
- Incremental loading
- High water marks
- ROW_NUMBER
- Idempotency
The rest of this case study is locked
The data, every step with its code and result, and the final answer open with a pass. A few case studies stay free so you can see how they work.