Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Partitioning & Pruning

What is predicate pushdown in Spark?

Mediumpredicate-pushdownparquetoptimization

Interview question

What is predicate pushdown in Spark?

How to answer

The short answer

means Spark pushes filter from the query plan down into the data source reader, so less data is decoded and shipped to executors.

Examples

  • Parquet/ORC: row-group / page-level filtering using column stats (min/max)
  • JDBC sources: generate SQL WHERE clauses executed in the database
  • filters: skip directories ( is a form of pushdown related to layout)
Output
Logical filter(amount > 100)
        |
   Physical plan pushes residual/predicate into Parquet scan
        |
   Reader skips non-matching row groups when stats allow
Python
# These filters can be pushed into Parquet scan
orders = spark.read.parquet("/lake/orders")
orders.filter("amount > 100 AND status = 'PAID'").select("order_id", "amount")

What blocks pushdown

  • Wrapping columns in opaque Python before filtering
  • Some complex expressions casted in ways the source cannot understand
  • Reading non-splittable / non-columnar formats (plain gzip CSV has weaker pruning)

Interview tip

"Push filters and column projections as close to the source as possible; keep data in columnar formats so pushdown and pruning actually work."

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill