Cloud · AWS

S3 data lake partitioning, prefix design, and small files

awsEasyawss3data-lakepartitioningperformance

Interview question

How should you design S3 prefix naming and partitioning for a data lake to optimize both Athena queries and writer throughput?

How to answer

The short answer

Designing Amazon S3 prefix structures and file sizes properly is foundational to building high-performance, cost-effective data lakes that query engines like Athena and Spark can scan efficiently.

Hive-style partitioning layout

Query engines like Athena, EMR, and Glue scan much faster when data is organized into Hive-compatible prefixes:

Output
s3://my-lake-bucket/curated/orders/
  |-- year=2026/month=10/day=16/part-000.parquet
  |-- year=2026/month=10/day=17/part-000.parquet
  `-- year=2026/month=10/day=17/part-001.parquet

When an Athena query includes WHERE year = 2026 AND month = 10 AND day = 17, the query engine prunes all other prefix paths, scanning only the relevant partition files and saving both scan time and query cost.

Partition cardinality and small files

Two common design traps degrade data lake performance on S3:

  • Over-partitioning: Partitioning by high-cardinality keys like customer_id or timestamp down to the minute creates thousands of subdirectories containing tiny kilobyte-sized files.
  • The : Querying thousands of small files forces query engines to spend more time negotiating S3 GET metadata requests than reading actual data. Target optimal file sizes between 128 megabytes and 512 megabytes.

Teams resolve the small files problem by running scheduled compaction jobs using AWS Glue or Spark that read daily partition folders and rewrite small files into consolidated Parquet blocks.