Cloud · AWS

Athena partition projection

awsMediumawsathenapartition-projectionglues3performance

Interview question

What is partition projection in Amazon Athena, and when would you use it instead of registering partitions in the Glue Data Catalog?

How to answer

The short answer

A partitioned Athena table normally keeps one catalog entry per in the AWS Glue . For a table partitioned by date and hour, that is thousands of entries, and each query first asks the catalog for them (a GetPartitions call) before it can decide which folders to read. With many partitions that lookup becomes the slow part.

Partition projection removes that lookup. Instead of storing partitions in the catalog, you describe their pattern in table properties and Athena works out the partition values and S3 locations itself, in memory.

Output
Without projection: query -> Glue catalog (list all partitions) -> prune -> read S3
With projection:    query -> compute partition values from table properties -> read S3

You switch it on with table properties, set per partition column:

Output
'projection.enabled' = 'true'
'projection.dt.type' = 'date'
'projection.dt.range' = '2026/01/01,NOW'
'projection.dt.format' = 'yyyy/MM/dd'
'storage.location.template' = 's3://my-bucket/events/${dt}/'

Supported column types include integer, date, enum (a fixed list such as region codes) and injected. A second benefit is that new partitions need no registration. With a plain Glue table you must add each new day's partition yourself, through a crawler, MSCK REPAIR TABLE or ALTER TABLE ADD PARTITION. With a date range ending at NOW, tomorrow's data is queryable automatically.

Trade-offs to mention

  • Athena ignores partition metadata in the catalog for that table once projection is on.
  • Projection only applies to queries through Athena. Redshift Spectrum, EMR and Athena for Spark read the normal catalog metadata.
  • If a projected partition does not exist in S3, Athena does not fail. It returns no rows. If more than half of the projected partitions are empty, AWS recommends traditional partitions.
  • Queries for values outside the configured range also return zero rows without an error, which can hide a wrong date.

Rule of thumb: use it for predictable layouts such as date folders or log data with very many partitions. Skip it when partitions are irregular.