Intermediate · 30 min · 5 steps
Re-running two years of history one day at a time, safely
Tarun Varma, Senior Data Operations Engineer at BharatFin Wealth
It is 8:30 AM on a Thursday. Tarun Varma, a data operations engineer at BharatFin Wealth, is paged by the site reliability team. BharatFin is an investment app that manages more than ₹35,000 crore for 2.8 million customers. The database copy that powers the customer dashboard has crashed because it ran out of memory.
The question
How can Python split a long date range into daily chunks and remember which days are done, so a big re-run never crashes the database and can restart from where it stopped?
- Backfills
- Date chunking
- Idempotence
- Airflow execution
The rest of this case study is locked
The data, every step with its code and result, and the final answer open with a pass. A few case studies stay free so you can see how they work.