Deepak Bhatia

@deepakbhatia

Free member

Learning Airflow while contributing to open source around lakehouse.

Chandigarh, PunjabStudent / switching in1 questions1 answers

AirflowDatabricksPostgres

Streak: 1d
XP: 643
Lessons done: 5
Studio passes: 4

First LessonFirst Ticket500 XP

Share on X Share on LinkedIn

Questions

votes

answers

views

Spark join on user_id skews badly, and broadcast didn't help

Processing ~500GB daily events joined to a 2M row dimension on user_id. One join key has ~8% of traffic. I tried broadcast(dim) but driver memory spiked. Salting sounds right but I haven't implemented it in production before. What pattern do you use?

PySparkpysparkskew