Broadcast Variable

PySpark data engineering interview problem. Difficulty: advanced. Pattern: Broadcast Join. About 20 minutes. Part of the Pro drill bank.

Use F.broadcast join instead of a broadcast dict + UDF lookup. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

When you join a big table to a small one, Spark can send a copy of the small table to every machine (a broadcast join). This avoids moving the big table across the network. You may also hear about broadcast variables, which share a Python object with all tasks, but they are used with UDFs. A broadcast join is usually the better choice. Join orders to F.broadcast(users) on user_id with an inner join. Select order_id, user_id, amount and name, and sort by order_id. Do not use broadcast variables or Python UDFs. The editor creates both tables and ends with result = orders. Change the last line to build your answer.

Requirements

• Broadcast the small users table with F.broadcast.
• Use an inner join on user_id.
• Select order_id, user_id, amount, name.
• Sort by order_id and store the DataFrame in result.

Constraints

• users is the small side, and orders is the big side.
• No UDFs, and no broadcast variables from Python.
• Every order in the sample has a matching user, so the result has 4 rows.

Examples

Input: orders + users Output: order_id | user_id | amount | name 101 | 1 | 50.0 | Ada 102 | 2 | 75.0 | Alan 103 | 1 | 20.0 | Ada 104 | 3 | 40.0 | Grace Broadcast join replaces dict UDF lookups.

Topics: lakebench, pyspark, broadcast.

More PySpark interview questions · All interview problems · Learn data engineering