Solve “UDF Performance Trap”

Free. Runs in your browser, no install.

PySpark · UDFs & Streaming

What is a Pandas UDF?

Hardpandas-udfarrowudfperformance

Interview question

What is a Pandas UDF (vectorized UDF) in PySpark?

How to answer

The short answer

A Pandas (vectorized UDF) applies Python logic to Arrow-backed batches of data as pandas.Series / DataFrame, instead of 1 row at a time. It is usually much faster than classic Python UDFs.

Example (Series to Series)

Python
import pandas as pd
from pyspark.sql.functions import pandas_udf

@pandas_udf("double")
def add_tax(s: pd.Series) -> pd.Series:
    return s * 1.18

df.withColumn("amount_taxed", add_tax("amount"))

Types (conceptual)

Output
Series -> Series      map-style column transform
Series -> Scalar      aggregation-style
Iterator variants     streaming batches for large data
Grouped map/agg       split-apply-combine with pandas

Diagram

Output
Spark column batches --Arrow--> pandas.Series --> your vectorized code --> Arrow --> Spark

Caveats

  • Still Python-side: slower than pure JVM built-ins
  • Watch memory: large batches + pandas copies
  • / return types must match Spark schema
  • Some operations still better as native Spark expressions

Interview tip

Contrast classic UDF (row) vs Pandas UDF (batch) vs native functions (best).

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill