intermediate

Small File Problem

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how creating thousands of tiny files hurts read performance, and how to compact small files using coalesce, repartition, or auto-compaction.

What You Will Learn

In this lesson, you will learn:
  • Metadata Overhead: Why reading many small files is slow.
  • Compaction: Merging small files into larger target sizes.
  • Configuration limits: Setting auto-compaction options in modern table formats.

Detailed Concept Explanation

The Small File Problem occurs when a Spark job writes thousands of tiny files (often under 1MB) to disk instead of a few large, well-sized files (target 128MB-512MB).

Why are Small Files bad?

  1. Metadata Bloat: File systems (like HDFS or cloud storage) must track metadata for every file. Having millions of files overloads the filesystem namespace.
  2. Read Latency: Opening and closing files takes time. Reading 1,000 files of 1MB takes significantly longer than reading a single 1GB file.

How it happens in Spark

If you write a DataFrame using a high number of partitions (e.g. 200 partitions) or write data partitioned by many nested directories, Spark writes a separate file for each partition in each folder.

How to resolve it

  • Coalesce: Call df.coalesce(num_files) before writing to disk to reduce the partition count without shuffles.
  • Delta Optimization: If using Delta Lake, run the OPTIMIZE command to automatically compact small files in the table.

Code Examples

Python (PySpark) Implementation (Compacting files before write)

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("SmallFiles").getOrCreate()

df = spark.range(1, 100000).repartition(200)

# Writing with 200 partitions creates 200 small files.
# Use coalesce to write exactly 2 files instead (avoiding shuffles).
df.coalesce(2).write.format("parquet").mode("overwrite").save("compacted_data")

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
range
repartition(200)
coalesce(2)
write.format(parquet)
save()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("SmallFilesScala").getOrCreate()

val df = spark.range(1, 100000).repartition(200)

df.coalesce(2).write.format("parquet").mode("overwrite").save("compacted_data")

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.