File formats & storage · Extra High-Value

What is partition pruning?

Mediumpartition pruningHive partitionsSparkfilters

Interview question

What is partition pruning?

How to answer

The short answer

means the query engine skips entire storage that cannot match the filter. If data lives under dt=... directories (or equivalent table-format partitions), a filter on dt avoids listing and reading the rest.

Python
df = spark.read.parquet("/lake/events")
df.filter(df.dt == "2024-01-02").count()

Requirements

  • Partition columns must be known to the metastore / path / table metadata
  • Filters must be selective and recognizable (avoid wrapping partition cols in opaque )
  • Dynamic partition pruning can prune a using values discovered from a dimension at runtime
  • Partition pruning: skip directories / partitions
  • : skip row groups/pages *inside* files
  • : skip unused columns

Interview tip: Always give a dated path example, then say pruning only helps if you filter on the partition columns.