Memory Manager

Overview

Learn how Spark's Unified Memory Manager allocates JVM executor memory dynamically between execution tasks and cache storage.

What You Will Learn

In this lesson, you will learn:

Unified Memory Manager: Coordinating execution and storage pools.
Dynamic Borrowing: How storage and execution borrow space from each other.
Eviction Priority: Why execution memory takes priority over storage cache memory.

Detailed Concept Explanation

Spark's Unified Memory Manager manages memory allocation inside the JVM executor processes.

Dynamic Memory Sharing

The Spark Memory pool (typically 60% of JVM heap) is divided into Execution Memory (shuffles/joins) and Storage Memory (cached partitions/broadcasts).

Flexible boundary: The boundary between execution and storage is dynamic. Storage memory can expand to use free execution memory, and execution memory can expand to use free storage memory.
Eviction Priority: If execution tasks require more memory, execution memory will evict cached partitions from storage memory to disk. However, storage memory cannot evict execution memory; it must wait until tasks complete and free up space.

Code Examples

Python (PySpark) Configuration

python

from pyspark.sql import SparkSession

# Tune JVM executor memory ratios
spark = SparkSession.builder \
    .appName("MemoryManagerInternals") \
    .config("spark.memory.fraction", "0.6") \
    .config("spark.memory.storageFraction", "0.5") \
    .getOrCreate()

df = spark.range(1, 100)
df.collect()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

config(spark.memory.fraction

0.6)

range

collect()

Scala Configuration

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("MemoryManagerScala")
  .config("spark.memory.fraction", "0.6")
  .config("spark.memory.storageFraction", "0.5")
  .getOrCreate()

val df = spark.range(1, 100)
df.collect()