Handling NULLs in PySpark

NULL handling shows up in filters, joins, aggregations, and math. Spark SQL NULL semantics mirror SQL: comparisons with NULL yield unknown, and you need null-safe helpers.

Common patterns

from pyspark.sql import functions as F

# Detect / filter
df.filter(F.col("user_id").isNull())
df.filter(F.col("user_id").isNotNull())

# Fill
df.na.fill({"amount": 0.0, "status": "UNKNOWN"})
df.fillna({"amount": 0.0})

# Drop
df.na.drop(subset=["user_id"])          # drop if user_id null
df.dropna(how="any")                    # any null in row
df.dropna(how="all", subset=["a", "b"]) # both null

# Replace in expressions
df.withColumn("amount", F.coalesce(F.col("amount"), F.lit(0.0)))
df.withColumn("name", F.when(F.col("name").isNull(), "n/a").otherwise(F.col("name")))

# Null-safe equality (important for joins)
df1.join(df2, df1.id.eqNullSafe(df2.id), "inner")

Gotchas

filter(col == 1)      drops rows where col is NULL (unknown is not true)
outer joins           introduce NULLs on the non-matching side
sum/avg               ignore NULLs; count(col) ignores NULLs; count(*) does not

Interview tip

Separate "missing because source omitted it" from "missing because left join did not match," and choose fill vs drop based on the grain and SLA.