Small File Problem
Overview
Learn how creating thousands of tiny files hurts read performance, and how to compact small files using coalesce, repartition, or auto-compaction.
What You Will Learn
In this lesson, you will learn:
- Metadata Overhead: Why reading many small files is slow.
- Compaction: Merging small files into larger target sizes.
- Configuration limits: Setting auto-compaction options in modern table formats.
Detailed Concept Explanation
The Small File Problem occurs when a Spark job writes thousands of tiny files (often under 1MB) to disk instead of a few large, well-sized files (target 128MB-512MB).
Why are Small Files bad?
- Metadata Bloat: File systems (like HDFS or cloud storage) must track metadata for every file. Having millions of files overloads the filesystem namespace.
- Read Latency: Opening and closing files takes time. Reading 1,000 files of 1MB takes significantly longer than reading a single 1GB file.
How it happens in Spark
If you write a DataFrame using a high number of partitions (e.g. 200 partitions) or write data partitioned by many nested directories, Spark writes a separate file for each partition in each folder.
How to resolve it
- Coalesce: Call
df.coalesce(num_files)before writing to disk to reduce the partition count without shuffles. - Delta Optimization: If using Delta Lake, run the
OPTIMIZEcommand to automatically compact small files in the table.
Code Examples
Python (PySpark) Implementation (Compacting files before write)
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("SmallFiles").getOrCreate()
df = spark.range(1, 100000).repartition(200)
# Writing with 200 partitions creates 200 small files.
# Use coalesce to write exactly 2 files instead (avoiding shuffles).
df.coalesce(2).write.format("parquet").mode("overwrite").save("compacted_data")
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("SmallFilesScala").getOrCreate()
val df = spark.range(1, 100000).repartition(200)
df.coalesce(2).write.format("parquet").mode("overwrite").save("compacted_data")
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.