Intermediate · 30 min · 4 steps
Removing duplicate events that were sent more than once
Kavita Rao, Lead Streaming Platform Engineer at StreamKart Logistics
It is Monday, 8:15 AM. Kavita Rao is a senior engineer on the streaming data platform at StreamKart Logistics, which moves over 800,000 parcels a day across India. She is checking the overnight reconciliation dashboard. StreamKart uses Change Data Capture (CDC): a tool called Debezium watches the main databases, and every time a row changes, it sends a message into Kafka, a system that carries data streams, so that other systems see the change.
The question
How do we find events that were delivered twice and keep exactly one copy of each, using a window function?
- Deduplication
- ROW_NUMBER
- Window functions
- CDC pipelines
The rest of this case study is locked
The data, every step with its code and result, and the final answer open with a pass. A few case studies stay free so you can see how they work.