Small File Problem

Overview

Learn how creating thousands of tiny files hurts read performance, and how to compact small files using coalesce, repartition, or auto-compaction.

What You Will Learn

Detailed Concept Explanation

The Small File Problem occurs when a Spark job writes thousands of tiny files (often under 1MB) to disk instead of a few large, well-sized files (target 128MB-512MB).

Why are Small Files bad?

1. Metadata Bloat: File systems (like HDFS or cloud storage) must track metadata for every file. Having millions of files overloads the filesystem namespace.
2. Read Latency: Opening and closing files takes time. Reading 1,000 files of 1MB takes significantly longer than reading a single 1GB file.

How it happens in Spark

If you write a DataFrame using a high number of partitions (e.g. 200 partitions) or write data partitioned by many nested directories, Spark writes a separate file for each partition in each folder.

How to resolve it

• Coalesce: Call df.coalesce(num_files) before writing to disk to reduce the partition count without shuffles.
• Delta Optimization: If using Delta Lake, run the OPTIMIZE command to automatically compact small files in the table.

Code Examples

Python (PySpark) Implementation (Compacting files before write)

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("SmallFiles").getOrCreate()

df = spark.range(1, 100000).repartition(200)

# Writing with 200 partitions creates 200 small files.
# Use coalesce to write exactly 2 files instead (avoiding shuffles).
df.coalesce(2).write.format("parquet").mode("overwrite").save("compacted_data")

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("SmallFilesScala").getOrCreate()

val df = spark.range(1, 100000).repartition(200)

df.coalesce(2).write.format("parquet").mode("overwrite").save("compacted_data")

Related Topics

• Repartition vs Coalesce
• Delta Lake
• Partitions