29
Worth measuring the serialized size before choosing broadcast.
This worked in dev on sample data but fails at full volume. Details:
Context: Coalesce vs repartition before write, still confused
Happy to share schema snippets or metrics if useful.
Worth measuring the serialized size before choosing broadcast.
In simple terms: your job is asking for way more data than it actually needs, and that extra data has to travel across the network or between machines, which is slow. Ask for less, or ask smarter.
Sign in to reply.