PySpark practice

PySpark join practice questions

5 problems, ordered from easiest. They run in your browser, and 5 are free.

The idea in a minute

A PySpark join matches rows from two DataFrames on a column. The how argument chooses what survives: inner keeps only matches, left keeps every row from the left side.

Joins are where big jobs get slow, so the problems also ask you to think about row counts and duplicate columns.

Example: Customers with and without orders

from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").config("spark.ui.enabled","false").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
customers = spark.createDataFrame([(1,"Asha"),(2,"Ravi"),(3,"Meera")],["customer_id","name"])
orders = spark.createDataFrame([(10,1,450),(11,1,150),(12,2,300)],["order_id","customer_id","amount"])
(customers.join(orders, "customer_id", "left")
          .groupBy("name").agg(F.count("order_id").alias("orders"))
          .orderBy("name").show())

What it prints

+-----+------+
| name|orders|
+-----+------+
| Asha|     2|
|Meera|     0|
| Ravi|     1|
+-----+------+

What to watch for

Join on a list of column names, not an equality expression, to avoid two copies of the key column.
Count the rows before and after to catch a join that repeats rows.
Left and right matter. The left DataFrame is the one you call join on.

Joins problems

Want a guided order across every topic? Open the practice sheets or the PySpark hub.