Weekly reprocessing job OOMs executors, but the identical code runs fine daily. The only difference is 7x the data volume for the weekly run.
Executor memory is already generous at 16GB. GC logs show long full-GC pauses right before the crash.
Accepted answer
7x volume with the same partition count often surfaces skew that was invisible at smaller scale. Check the Spark UI's task duration distribution for the failing stage, a few tasks running 50x longer than the median is the smoking gun.
Sign in to reply.
