advanced

Memory Manager

By DataPlayArena Technical Review BoardUpdated 2026-07-0910 min readRuntime Verified

Overview

Learn how Spark's Unified Memory Manager allocates JVM executor memory dynamically between execution tasks and cache storage.

What You Will Learn

In this lesson, you will learn:
  • Unified Memory Manager: Coordinating execution and storage pools.
  • Dynamic Borrowing: How storage and execution borrow space from each other.
  • Eviction Priority: Why execution memory takes priority over storage cache memory.

Detailed Concept Explanation

Spark's Unified Memory Manager manages memory allocation inside the JVM executor processes.

Dynamic Memory Sharing

The Spark Memory pool (typically 60% of JVM heap) is divided into Execution Memory (shuffles/joins) and Storage Memory (cached partitions/broadcasts).

  • Flexible boundary: The boundary between execution and storage is dynamic. Storage memory can expand to use free execution memory, and execution memory can expand to use free storage memory.
  • Eviction Priority: If execution tasks require more memory, execution memory will evict cached partitions from storage memory to disk. However, storage memory cannot evict execution memory; it must wait until tasks complete and free up space.

Code Examples

Python (PySpark) Configuration

python
from pyspark.sql import SparkSession

# Tune JVM executor memory ratios
spark = SparkSession.builder \
    .appName("MemoryManagerInternals") \
    .config("spark.memory.fraction", "0.6") \
    .config("spark.memory.storageFraction", "0.5") \
    .getOrCreate()

df = spark.range(1, 100)
df.collect()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
config(spark.memory.fraction
0.6)
range
collect()

Scala Configuration

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("MemoryManagerScala")
  .config("spark.memory.fraction", "0.6")
  .config("spark.memory.storageFraction", "0.5")
  .getOrCreate()

val df = spark.range(1, 100)
df.collect()

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.