Burnout from constant firefighting, how did you actually fix your pipelines' reliability?

Worth measuring the serialized size before choosing broadcast.

Doing the root-cause audit now. Already obvious that one flaky vendor feed accounts for a disproportionate share of our pages.

Add data quality checks and contracts at ingestion boundaries so bad data fails loudly and immediately at the source, rather than silently propagating and causing a confusing downstream failure hours or days later that takes much longer to diagnose.

Categorize every fire by root cause for a month before fixing anything reactively. In our case it revealed that about 60% of incidents traced back to just two brittle upstream sources, and fixing those two structurally instead of the symptom each time freed up most of our firefighting time.