Aditya Bhat

Pro

@adityabhat

Pro member

Consultant helping teams ship reliable Flink workloads.

Coimbatore, Tamil Nadu0–2 years1 questions5 answers

FlinkdbtKafka

Streak: 13d
XP: 5658
Lessons done: 47
Studio passes: 33

Week Warrior25 Lessons25 Tickets5,000 XP

Share on X Share on LinkedIn

Questions

votes

answers

views

toPandas() OOMs the driver on a DataFrame that should easily fit

Calling df.toPandas() on what I estimated as a 2GB DataFrame kills the driver with an OOM. Driver has 8GB. The estimate was rows times avg_row_size, but that's clearly wrong somewhere.

PySparkpysparkdriver