File formats & storage · Performance & Layout

The small files problem

Mediumsmall filescompactionSparkS3performance

Interview question

What is the small-files problem in data lakes?

How to answer

The short answer

The small-files problem is when a dataset is stored as a huge number of tiny files. Engines and object stores then spend more time on listing, opening, planning, and task scheduling than on reading useful bytes.

Common causes

  • Writing with far too many output (repartition(10000) on modest data)
  • Streaming micro-batches each flushing many files
  • Over-partitioning (partition by hour *and* high-cardinality columns)
  • Frequent appends without compaction

Mitigations

  • coalesce / repartition to a target file count or size before write
  • Raise maxRecordsPerFile thoughtfully
  • Schedule compaction (Delta OPTIMIZE, Iceberg rewrite, Hudi compact)
  • Prefer coarser partitions for small daily volumes

Interview tip: Connect small files to cloud LIST costs, slow job starts, and why lakehouse formats ship compaction utilities. Target roughly hundreds-of-MB files unless your engine docs say otherwise.