When a team gets 200 alerts a day and ignores them, the alerts have failed, because their only job is to make someone act. The fix is fewer, better alerts. Start by looking at what the alerts really are.
Audit the noise
Export a month of alerts and group them by type. You will probably find that a small number of alert types account for most of the volume: task retries that succeed on the second try, known flaky sources, warnings with no action, duplicates of the same incident from several systems. For each type, ask a simple question: when this fires, does someone need to do something? If the answer is no, it should not page anyone.
Alert on symptoms that need action
- Page for things that hurt: an SLA is going to be missed, a critical table is stale, a data quality check on a key table failed, a pipeline is stuck.
- Do not alert on each retry. Alert if the final attempt fails, or retries are unusually many.
- Prefer one alert per incident, not one per failing task. If the upstream table is late, forty downstream tasks failing is one problem. Group them, or suppress downstream alerts when the upstream one is firing.
Severity and routing
Use levels. A critical alert pages the on-call person at night. A warning creates a ticket or goes to a channel that is reviewed during working hours. Info goes to a dashboard only. Route each alert to the team that owns the pipeline, using ownership metadata, and not to one big shared channel that everyone mutes.
Make each alert useful
Include what failed, since when, the impact (which dashboards or SLAs), a link to the logs, and a link to the runbook with the first things to try. An alert that says "task failed" wastes the first ten minutes of every incident.
Keep it healthy
- Review alerts regularly, say every two weeks. Delete or tune any that fired often with no action.
- Track the number of alerts per on-call shift and how many needed action. Make that ratio a team goal.
- When an incident is not caught by alerts, add one. When an alert is noisy, fix the cause or remove it.
The target is simple: when an alert fires, people trust that it matters.