Broadcast Join

PySpark data engineering interview problem. Difficulty: beginner. Pattern: Broadcast Join. About 12 minutes. Free to practice.

Join orders to a small users table with F.broadcast. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

When one table is big and the other is small, Spark can send a copy of the small table to every machine. Then the big table does not need to be moved around the network (shuffled). This is called a broadcast join, and it is a common way to make joins faster. Treat users as the small table. Join orders to F.broadcast(users) on user_id with an inner join. Select order_id, user_id, amount, name and email, and sort by order_id. The editor creates orders and users and ends with result = orders. Change the last line so result holds your answer. The simulator prints a note about the broadcast. On a real cluster the hint changes how the join runs, but not what it returns.

Requirements

• Wrap the small table in F.broadcast(...).
• Use an inner join on user_id.
• Select order_id, user_id, amount, name, email.
• Sort by order_id and store the DataFrame in result.

Constraints

• users is the small table. Broadcasting the big table would defeat the point.
• The join type is inner, so unmatched orders are dropped.
• A broadcast changes the execution plan, not the result of the join.

Examples

Input: orders + small users Output: order_id | user_id | amount | name | email 101 | 1 | 50.0 | Ada | [email protected] 102 | 2 | 75.0 | Alan | [email protected] 103 | 3 | 20.0 | Grace | [email protected] 105 | 1 | 60.0 | Ada | [email protected] Broadcast avoids shuffling the large side in production; result matches an inner join.

Topics: lakebench, pyspark, broadcast.

More PySpark interview questions · All interview problems · Learn data engineering