Intermediate · 30 min · 4 steps
Finding the one stuck data stream that an average hides
Naveen Sundaram, Real-Time Infrastructure Engineer at TeleMetrics Cloud
It is Wednesday, 9:15 AM. Naveen Sundaram is an infrastructure engineer at TeleMetrics Cloud, a platform that takes in over 80,000 sensor readings every second from electric vehicle fleets, charging stations and grid batteries. He is watching the streaming operations screen.
The question
How do we work out how far behind each Kafka partition is, in Python, and raise alerts and scaling advice from the worst one?
- Kafka
- Consumer lag
- Streaming offsets
- Alerting thresholds
The rest of this case study is locked
The data, every step with its code and result, and the final answer open with a pass. A few case studies stay free so you can see how they work.