Solve “UDF Performance Trap”

Free. Runs in your browser, no install.

PySpark · UDFs & Streaming

What is a UDF?

Hardudfperformancepython

Interview question

What is a UDF in PySpark, and what are the downsides?

How to answer

The short answer

A (User Defined Function) is custom code registered so Spark can apply it row-wise (or via SQL) when built-in functions are not enough.

Python UDF example

Python
from pyspark.sql.functions import udf
from pyspark.sql.types import StringType

def bucket(amount):
    if amount is None:
        return "UNK"
    return "HIGH" if amount >= 100 else "LOW"

bucket_udf = udf(bucket, StringType())
df.withColumn("bucket", bucket_udf("amount"))

Why they hurt performance

Output
Catalyst/Tungsten optimized ops   vs   Python UDF
  whole-stage codegen                    row-by-row
  JVM execution                          Python worker process + serialization

Python UDFs break many optimizations, add serialization between JVM and Python, and prevent whole-stage codegen from fusing operators.

Prefer

  • Built-in pyspark.sql.functions
  • SQL expressions
  • when/otherwise, higher-order functions on arrays
  • Pandas UDFs / Spark Connect vectorized paths when custom logic is required

Interview tip

"UDFs are an escape hatch. I reach for them last and prefer Pandas UDFs or native functions when custom logic is unavoidable."

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill