Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

filter() vs where()

Mediumfilterwheredataframe-api

Interview question

What is the difference between filter() and where() in PySpark?

How to answer

The short answer

In PySpark , `filter()` and `where()` are aliases. They do the same thing: keep rows that satisfy a boolean condition.

Python
from pyspark.sql import functions as F

df.filter(F.col("amount") > 100)
df.where(F.col("amount") > 100)

df.filter("amount > 100")   # SQL expression string also works
df.where("amount > 100")

Why both exist

where reads more like SQL for analysts; filter matches RDD/functional naming. Teams usually pick one style and stick to it.

Diagram

Output
DataFrame rows ---- condition ----> subset of rows
                 filter / where
                      (same)

Do not confuse DataFrame filter with RDD filter, or with select (columns vs rows). Also remember NULL semantics: rows where the is unknown are dropped.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill