The small file problem

Why small files hurt

Reading a file has a fixed cost that does not depend on its size:

On object stores like S3, ADLS and GCS, request latency, not bandwidth, dominates. Reading 10 GB as 100,000 files can take ten times longer than the same 10 GB as 80 files.

How Spark jobs create them

1. Too many shuffle partitions at write time. A write after a join or aggregation produces one file per non-empty partition. 200 partitions writing 50 MB of data gives 200 files of 250 KB.
2. partitionBy on a high-cardinality column. Each task writes one file per partition value it holds. 200 tasks × 365 dates = up to 73,000 files per run.
3. Streaming micro-batches. A stream that commits every 10 seconds writes files every 10 seconds: 8,640 batches a day, each with several files.
4. Frequent small appends. Hourly or per-event loads of a few MB each.
5. Over-partitioned sources. Reading thousands of files and writing straight back preserves the count.

Walk through one bad write

daily.write.mode("append").partitionBy("event_date", "country").parquet(path)

daily.repartition("event_date", "country") \
    .write.mode("append").partitionBy("event_date", "country").parquet(path)

INSERT INTO events_by_country
SELECT /*+ REPARTITION(event_date, country) */ *
FROM daily

Preventing them when writing

Delta Lake can do this for you: optimized writes add an adaptive shuffle before writing so each partition gets fewer, larger files, and auto compaction runs a small compaction after a write. They are table properties (delta.autoOptimize.optimizeWrite, delta.autoOptimize.autoCompact) on Databricks and in recent open-source Delta releases; check your version.

Fixing them afterwards

# Plain Parquet: rewrite one partition into fewer files, then swap it in
(spark.read.parquet(f"{path}/event_date=2025-03-01")
    .repartition(8)
    .write.mode("overwrite").parquet(f"{tmp}/event_date=2025-03-01"))

# Delta Lake
from delta.tables import DeltaTable
DeltaTable.forPath(spark, path).optimize().executeCompaction()

-- Delta Lake
OPTIMIZE events WHERE event_date >= '2025-03-01';

-- Apache Iceberg
CALL catalog.system.rewrite_data_files(table => 'db.events');

With plain Parquet, rewriting in place is unsafe for concurrent readers: they can see the old files deleted before the new ones appear. Table formats make compaction a single atomic commit, which is one of their biggest practical benefits. Old files remain until VACUUM (Delta) or expire_snapshots (Iceberg) removes them.

Detecting the problem

• DESCRIBE DETAIL table on Delta returns numFiles and sizeInBytes: divide to get the average file size. Under about 32 MB is worth fixing.
• For Iceberg, query the metadata table db.events.files for file_size_in_bytes.
• In the Spark UI, a scan with tens of thousands of tasks reading a few hundred KB each, or a long gap before the first job (planning and listing), points to small files.

Common mistakes