Batch & Streaming · Batch Processing

The small files problem

Mediumsmall filesParquetcompactionlake

Interview question

What is the small files problem in data lakes, and how do you fix it?

How to answer

The short answer

Small files means a table or path is made of thousands/millions of tiny files (KB–few MB) instead of fewer larger columnar files. Listing, opening, and planning over them kills job time and stresses the metastore/object store.

How they appear

  • Streaming sinks writing every micro-batch
  • repartition too high before write
  • Many parallel tasks each writing 1 file
  • Over-partitioned Hive-style paths (date=.../hour=.../minute=...)
Output
Bad:  day/  f1(20KB) f2(20KB) ... f50000(20KB)
Good: day/  part-00001.parquet (256MB) part-00002.parquet (256MB)

Fixes

  • / before write to target file size (~128–512MB Parquet often)
  • Compaction jobs (Delta OPTIMIZE, Iceberg rewrite_data_files)
  • Fewer ; batch micro-batch outputs
  • Auto-compact features in lakehouse formats

Interview tip: Link cause (too many writers/partitions) → symptom (slow LIST + task startup) → fix (compaction / coalesce).