PySpark practice

PySpark window functions practice questions

14 problems, ordered from easiest. They run in your browser, and 1 is free.

Start with Lag and Lead

The idea in a minute

A PySpark window function calculates across related rows but keeps each row. You describe the window with Window.partitionBy and orderBy, then use a function such as row_number or lag over it.

It is the DataFrame version of the SQL idea, so if you know OVER in SQL you already know the shape.

Example: Rank orders inside each city

from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.master("local[2]").config("spark.ui.enabled","false").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")
orders = spark.createDataFrame([(1,"Asha","Pune",450),(2,"Ravi","Pune",300),(3,"Meera","Delhi",900),(4,"Asha","Delhi",150),(5,"Ravi","Pune",250)],["order_id","customer","city","amount"])
from pyspark.sql import Window
w = Window.partitionBy("city").orderBy(F.col("amount").desc())
orders.withColumn("rank_in_city", F.rank().over(w)).orderBy("city", "rank_in_city").show()

What it prints

+--------+--------+-----+------+------------+
|order_id|customer| city|amount|rank_in_city|
+--------+--------+-----+------+------------+
|       3|   Meera|Delhi|   900|           1|
|       4|    Asha|Delhi|   150|           2|
|       1|    Asha| Pune|   450|           1|
|       2|    Ravi| Pune|   300|           2|
|       5|    Ravi| Pune|   250|           3|
+--------+--------+-----+------+------------+

What to watch for

  • Define the window once and reuse it for several columns.
  • You cannot filter on a window column in the same step. Add the column, then filter.
  • Add a tie-break column to orderBy so the result is the same every run.

Window Functions problems

  1. Lag and Leadintermediate · Free
  2. Rank vs Dense Rankintermediate · Locked
  3. Running Totalintermediate · Locked
  4. First and Last Valueintermediate · Locked
  5. Top N Per Groupintermediate · Locked
  6. Percentile Rankintermediate · Locked
  7. Rolling Averageintermediate · Locked
  8. Forward-fill readings per deviceintermediate · Locked
  9. Above department average and second-highest salaryintermediate · Locked
  10. Failure streak that resets on successintermediate · Locked
  11. Cumulative distinct users per dayintermediate · Locked
  12. Most common page paths per sessionintermediate · Locked
  13. Sessionize clickstreamadvanced · Locked
  14. Longest attendance streakadvanced · Locked

Want a guided order across every topic? Open the practice sheets or the PySpark hub.