File formats & storage · Formats & Table Formats

What is column pruning?

Easycolumn pruningprojectionParquetI/O

Interview question

What is column pruning (projection pushdown)?

How to answer

The short answer

(also called projection pushdown) means the engine reads only the columns required by the query, not every column in the file.

Problem first. A wide might have 80 columns. A dashboard needs order_id, amount, and region. In a row store or SELECT * CSV mindset, you still pay for the other 77 fields. In Parquet/ORC, those other column chunks can stay on disk.

Python
# Good: project early so the scan can prune columns
df = spark.read.parquet("/lake/orders").select("order_id", "amount", "region")

# Bad habit: pull everything then drop later
df = spark.read.parquet("/lake/orders")  # may still prune if plan is smart,
df = df.select("order_id")               # but SELECT * patterns and UDFs hurt

Why it matters financially

Less I/O, less CPU decompression, less memory, cheaper cloud scans.

Interview tip: "Column pruning = do not read unused columns." Contrast with (skip rows/chunks) and (skip directories).

Seeing it in a real file. I wrote a 200,000-row Parquet file with DuckDB: order_id, user_name, amount, region, and one long text column notes. Parquet stores each column in its own chunk, so the compressed size per column can be read from the file's metadata:

Output
amount       379,265 bytes    9.4%
notes      2,825,135 bytes   70.0%
order_id     801,181 bytes   19.9%
region         2,555 bytes    0.1%
user_name     26,305 bytes    0.7%
query needs amount+region: 381,820 of 4,034,441 bytes = 9.5%

A query that needs only amount and region reads about 9.5% of the file. The saving depends on how wide the skipped columns are: here notes was made deliberately long. A CSV cannot do this, because the engine must scan every line to find the fields.