Partition Pruning
Overview
Learn how Partition Pruning optimizes reading speed by skipping entire directory folders on disk based on partition filters.
What You Will Learn
In this lesson, you will learn:
- Directory Layouts: Folder structures mapped to partition keys.
- Static Pruning: Skipping directories during query compilation.
- Dynamic Pruning (DPP): Skipping partitions dynamically at runtime using join keys.
Detailed Concept Explanation
When you save data partitioned by a column (e.g. df.write.partitionBy("country").parquet("data")), Spark creates a nested directory structure on disk:
data/country=US/
data/country=FR/
Partition Pruning is the optimization where Spark skips reading folders that do not match the filter conditions in your query.
- Static Partition Pruning: If your query is
SELECT * FROM data WHERE country = 'US', Spark only scans files inside thecountry=US/directory, skipping all other folders. - Dynamic Partition Pruning (DPP): In a join query (e.g. joining a large sales table partitioned by date with a small date dimension table filtered to 'December'), Spark evaluates the join key at runtime and only reads the sales partitions that match the matching dates, avoiding scanning the entire sales table.
Code Examples
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("PartitionPruning").getOrCreate()
# Read partitioned directory
df = spark.read.parquet("data")
# Query with partition filter
result = df.filter(df.country == "US")
# PartitionFilters shows skipped directories in physical plan
result.explain()
Expected Output
== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet [id#0,country#1] Batched: true, Format: Parquet,
PartitionFilters: [isnotnull(country#1), (country#1 = US)], PushedFilters: [], ReadSchema: struct<id:int>
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("PruningScala").getOrCreate()
val df = spark.read.parquet("data")
val result = df.filter($"country" === "US")
result.explain()
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.