12
Accepted answer
Worth measuring the serialized size before choosing broadcast.
A topic doing maybe 500 messages/sec was set up with 100 partitions "for future scale." Now seeing longer rebalance times and more open file handles than expected on the brokers. Was that overprovisioning a mistake?
Accepted answer
Worth measuring the serialized size before choosing broadcast.
Reduced a newer topic doing similar volume to 12 partitions based on this. Rebalance times there are noticeably faster now.
Note that merge on Delta still needs unique keys defined correctly.
+1, saw identical behaviour after upgrading Spark 3.4 to 3.5.
Document the grain decision, most BI bugs turn out to be grain bugs.
Yes, generally. Each partition costs broker-side resources, file handles, replication overhead, memory, regardless of throughput, and more partitions also means longer consumer group rebalances. Size partitions for actual target throughput and consumer parallelism, not a hoped-for future.
Plain English: the system prefers to guess a good-enough plan than the perfect plan, because figuring out the perfect plan would take longer than just running the good-enough one.
Yes, the partial index was the win for us too.
Note that merge on Delta still needs unique keys defined correctly.
Sign in to reply.
© 2026 Lakebench, operated by Hunnurji Rao. Bengaluru, Karnataka, India.
No cluster. No install. Just the tab.