Intermediate · 30 min · 4 steps
Keeping bad records out of clean tables without losing them
Arjun Nambiar, Principal Data Quality Architect at FinPay Global
It is Tuesday, 8:30 AM. Arjun Nambiar is the principal data quality architect at FinPay Global, a company that handles over 5 million payments a day across South-East Asia and India, in several currencies. Data flows in from more than 40 partners. He is called to an emergency meeting with the Chief Risk Officer.
The question
How do we check every incoming record against a set of rules, send the good ones to the clean table, and keep the bad ones in a separate table with the reason?
- Data quality
- Medallion architecture
- Quarantine routing
- CASE expressions
The rest of this case study is locked
The data, every step with its code and result, and the final answer open with a pass. A few case studies stay free so you can see how they work.