Solve “Null count for every column”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

Handling NULLs in PySpark

Mediumnullsdataframedata-quality

Interview question

How do you handle NULLs in PySpark DataFrames?

How to answer

The short answer

NULL handling shows up in filters, joins, aggregations, and math. Spark SQL NULL semantics mirror SQL: comparisons with NULL yield unknown, and you need null-safe helpers.

Common patterns

Python
from pyspark.sql import functions as F

# Detect / filter
df.filter(F.col("user_id").isNull())
df.filter(F.col("user_id").isNotNull())

# Fill
df.na.fill({"amount": 0.0, "status": "UNKNOWN"})
df.fillna({"amount": 0.0})

# Drop
df.na.drop(subset=["user_id"])          # drop if user_id null
df.dropna(how="any")                    # any null in row
df.dropna(how="all", subset=["a", "b"]) # both null

# Replace in expressions
df.withColumn("amount", F.coalesce(F.col("amount"), F.lit(0.0)))
df.withColumn("name", F.when(F.col("name").isNull(), "n/a").otherwise(F.col("name")))

# Null-safe equality (important for joins)
df1.join(df2, df1.id.eqNullSafe(df2.id), "inner")

Gotchas

Output
filter(col == 1)      drops rows where col is NULL (unknown is not true)
outer joins           introduce NULLs on the non-matching side
sum/avg               ignore NULLs; count(col) ignores NULLs; count(*) does not

Interview tip

Separate "missing because source omitted it" from "missing because left join did not match," and choose fill vs drop based on the grain and .

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill