Shuffle Manager

Overview

Learn how Spark's Shuffle Manager coordinates sort-based shuffles, including how it writes partition files and fetches them across nodes.

What You Will Learn

In this lesson, you will learn:

Sort Shuffle Writer: How Spark aggregates and sorts partition blocks before writing.
Index files: Mapping partitions to file byte offsets.
External Shuffle Service: Keeping shuffle data alive during dynamic allocation.

Detailed Concept Explanation

The Shuffle Manager coordinates the transfer of data partitions between stages (shuffle write and shuffle read).

Sort Shuffle Mechanics

Spark uses Sort-Based Shuffle by default:

Shuffle Write: The executor groups records by key, sorts them in memory, and writes them to a single data file. It also writes a corresponding Index File mapping partition indices to byte offsets inside the data file.
Shuffle Read: Downstream executors read the index file to fetch only the specific byte ranges representing their target keys.

Code Examples

Python (PySpark) Implementation

python

from pyspark.sql import SparkSession

# Shuffle manager uses SortShuffleManager by default
spark = SparkSession.builder.appName("ShuffleManager").getOrCreate()

df = spark.range(1, 100).repartition(4)
result = df.groupBy("id").count()

result.collect()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

range

repartition(4) [SortShuffleWriter]

groupBy.count()

collect() [SortShuffleReader]

Scala Implementation

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ShuffleManagerScala").getOrCreate()

val df = spark.range(1, 100).repartition(4)
val result = df.groupBy("id").count()

result.collect()