Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Partitioning & Pruning

What is partition pruning?

Mediumpartition-pruningiooptimization

Interview question

What is partition pruning in Spark?

How to answer

The short answer

means Spark skips entire storage that cannot match the query filters. If data is laid out as dt=.../region=..., a filter on those columns can avoid reading most files.

Example

Output
Dataset layout:
  /events/dt=2024-01-01/...
  /events/dt=2024-01-02/...
  /events/dt=2024-01-03/...

Query: WHERE dt = '2024-01-02'
Spark reads only dt=2024-01-02 directories (partition pruning)
Python
df = spark.read.parquet("/lake/events")
df.filter(df.dt == "2024-01-02").count()

Requirements

  • Partition columns must be present in the path/metastore metadata
  • Filters must be selective and applied in a way Catalyst can push to the data source
  • Dynamic partition pruning can also prune partitions of a using values discovered from a dimension side at runtime (broadcast/DPP)

Interview tip

Pair with (read only needed Parquet columns) and for nested filters inside files.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill