Partition Pruning

Overview

Learn how Partition Pruning optimizes reading speed by skipping entire directory folders on disk based on partition filters.

What You Will Learn

Detailed Concept Explanation

When you save data partitioned by a column (e.g. df.write.partitionBy("country").parquet("data")), Spark creates a nested directory structure on disk: data/country=US/ data/country=FR/

Partition Pruning is the optimization where Spark skips reading folders that do not match the filter conditions in your query.

• Static Partition Pruning: If your query is SELECT * FROM data WHERE country = 'US', Spark only scans files inside the country=US/ directory, skipping all other folders.
• Dynamic Partition Pruning (DPP): In a join query (e.g. joining a large sales table partitioned by date with a small date dimension table filtered to 'December'), Spark evaluates the join key at runtime and only reads the sales partitions that match the matching dates, avoiding scanning the entire sales table.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PartitionPruning").getOrCreate()

# Read partitioned directory
df = spark.read.parquet("data")

# Query with partition filter
result = df.filter(df.country == "US")

# PartitionFilters shows skipped directories in physical plan
result.explain()

Expected Output

== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet [id#0,country#1] Batched: true, Format: Parquet, 
   PartitionFilters: [isnotnull(country#1), (country#1 = US)], PushedFilters: [], ReadSchema: struct<id:int>

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("PruningScala").getOrCreate()

val df = spark.read.parquet("data")
val result = df.filter($"country" === "US")

result.explain()

Related Topics

• Predicate Pushdown
• Reading Data
• Writing Data