advanced
Memory Manager
Overview
Learn how Spark's Unified Memory Manager allocates JVM executor memory dynamically between execution tasks and cache storage.
What You Will Learn
In this lesson, you will learn:
- Unified Memory Manager: Coordinating execution and storage pools.
- Dynamic Borrowing: How storage and execution borrow space from each other.
- Eviction Priority: Why execution memory takes priority over storage cache memory.
Detailed Concept Explanation
Spark's Unified Memory Manager manages memory allocation inside the JVM executor processes.
Dynamic Memory Sharing
The Spark Memory pool (typically 60% of JVM heap) is divided into Execution Memory (shuffles/joins) and Storage Memory (cached partitions/broadcasts).
- Flexible boundary: The boundary between execution and storage is dynamic. Storage memory can expand to use free execution memory, and execution memory can expand to use free storage memory.
- Eviction Priority: If execution tasks require more memory, execution memory will evict cached partitions from storage memory to disk. However, storage memory cannot evict execution memory; it must wait until tasks complete and free up space.
Code Examples
Python (PySpark) Configuration
python
from pyspark.sql import SparkSession
# Tune JVM executor memory ratios
spark = SparkSession.builder \
.appName("MemoryManagerInternals") \
.config("spark.memory.fraction", "0.6") \
.config("spark.memory.storageFraction", "0.5") \
.getOrCreate()
df = spark.range(1, 100)
df.collect()
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
config(spark.memory.fraction
0.6)
range
collect()
Scala Configuration
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.appName("MemoryManagerScala")
.config("spark.memory.fraction", "0.6")
.config("spark.memory.storageFraction", "0.5")
.getOrCreate()
val df = spark.range(1, 100)
df.collect()
Related Topics
DPA
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.