Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

select() vs withColumn()

Mediumselectwithcolumndataframe-api

Interview question

What is the difference between select() and withColumn() in PySpark?

How to answer

The short answer

Both reshape columns, but for different jobs.

select(...)

Builds a new column list. You keep only what you list (plus expressions). Great for projection and renaming in one pass.

withColumn(name, expr)

Adds or replaces one column and keeps all other columns. Convenient, but chaining many withColumn calls can create a deep logical plan.

Python
from pyspark.sql import functions as F

# select: project explicitly
out = df.select(
    "order_id",
    F.col("amount").alias("amt"),
    (F.col("amount") * 1.18).alias("amt_with_tax"),
)

# withColumn: add/replace one column
out2 = df.withColumn("amt_with_tax", F.col("amount") * 1.18)

# Many new columns: prefer one select (cleaner plan)
out3 = df.select(
    "*",
    (F.col("amount") * 1.18).alias("amt_with_tax"),
    F.upper("status").alias("status_u"),
)

Diagram

Output
select(a, b, expr)     -> keep only a, b, expr
withColumn("c", expr)  -> keep all prior columns + c

Interview tip

For of many columns, one select (or selectExpr) is often clearer and can be friendlier to the optimizer than 20 chained withColumns.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill