Keerthi Gowda

Pro

@keerthigowda

Pro member

SRE-turned-DE. Care about observability, cost, and GCP.

Chennai, Tamil Nadu5–10 years1 questions5 answers

GCPdbtAirflowRedshiftFlink

Streak: 17d
XP: 3903
Lessons done: 26
Studio passes: 26

Fortnight Streak25 Lessons25 Tickets2,500 XP

Share on X Share on LinkedIn

Questions

votes

answers

views

Executor OOM only under a specific weekly batch, turned out to be a skewed join key

Weekly reprocessing job OOMs executors, but the identical code runs fine daily. The only difference is 7x the data volume for the weekly run. Executor memory is already generous at 16GB. GC logs show long full-GC pauses right before the crash.

Apache Sparksparkexecutors