Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · Caching & Reliability

What is the small-files problem?

Hardsmall-filesiolakehouseperformance

Interview question

What is the small-files problem in Spark and data lakes?

How to answer

The short answer

The small-files problem is when a dataset is stored as a huge number of tiny files. Spark (and object stores) then spend more time on listing, opening, and scheduling than on reading bytes.

Causes

  • Writing with too many output (repartition(10000) on small data)
  • Streaming micro-batches each writing many files
  • Over-partitioning Hive partitions (partition by high-cardinality columns)
  • Frequent appends without compaction

Diagram

Output
Good:  20 x 512MB Parquet files
Bad:   50,000 x 8KB Parquet files  -> driver listing + task overhead explode

Mitigations

Python
# Reduce output writers before write
df.coalesce(20).write.mode("overwrite").parquet(path)

# Or repartition to a sensible target size
df.repartition(50).write.partitionBy("dt").parquet(path)

# Compact later (lakehouse OPTIMIZE / rewrite jobs)

Also: raise maxRecordsPerFile carefully, use , schedule compaction, prefer fewer partitions for small daily volumes.

Interview tip

Connect small files to cloud listing costs, NameNode pressure (HDFS), slow job startup, and why table formats add compaction utilities.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill