Repartition vs Coalesce

Overview

Learn the critical differences between repartition and coalesce to optimize partition layouts while minimizing network shuffles.

What You Will Learn

Detailed Concept Explanation

Managing partition layout is crucial for query efficiency. Spark provides two API methods to adjust partition counts:

1. repartition(n)

• Behavior: Performs a full network shuffle to redistribute data evenly across n new partitions.
• Direction: Can increase or decrease the partition count.
• Use Case: Balancing data sizes across partitions to avoid skew before joins or aggregations.
• Cost: High, due to network and disk I/O shuffles.

2. coalesce(n)

• Behavior: Decreases partition count without performing a full network shuffle. It merges adjacent partitions on the same executor node.
• Direction: Can only decrease the partition count.
• Use Case: Reducing partition counts to optimize file sizes before writing to disk.
• Cost: Low, as it avoids network shuffles.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("RepartitionCoalesce").getOrCreate()

df = spark.range(1, 1000).repartition(10)
print("Partitions after repartition:", df.rdd.getNumPartitions())

# Reduce partitions using coalesce (no shuffle)
coalesced_df = df.coalesce(2)
print("Partitions after coalesce:", coalesced_df.rdd.getNumPartitions())

Expected Output

Partitions after repartition: 10
Partitions after coalesce: 2

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("RepartitionCoalesceScala").getOrCreate()

val df = spark.range(1, 1000).repartition(10)
println(s"Partitions after repartition: ${df.rdd.getNumPartitions}")

val coalesced = df.coalesce(2)
println(s"Partitions after coalesce: ${coalesced.rdd.getNumPartitions}")

Interview Perspective

Related Topics

• Partitions
• Shuffle
• Small File Problem