Memory Management

Overview

Learn how Spark manages memory inside JVM executors, including how it divides memory between execution, storage, and user spaces.

What You Will Learn

In this lesson, you will learn:

JVM Heap Layout: How executor memory is structured.
Execution vs Storage Memory: Dynamic boundary sharing.
Off-heap Memory: Bypassing JVM garbage collection limits.

Detailed Concept Explanation

Each Spark Executor runs as a single JVM process. The total JVM heap memory is divided into three main pools:

text

+-------------------------------------------------------------+
|                     JVM Executor Heap                       |
+------------------------------+------------------------------+
|         Spark Memory         |         User Memory          |
|  (Execution vs Storage 60%)  |            (40%)             |
+------------------------------+------------------------------+

Memory Pools

Spark Memory (60% by default):
- Execution Memory: Used to store intermediate states for shuffles, joins, and aggregations.
- Storage Memory: Used to cache data (df.cache()) and broadcast variables.
- Dynamic Sharing: If no cached data is using Storage Memory, Execution Memory can grow to use 100% of the Spark Memory pool, and vice versa. If execution memory needs space, it can evict cached storage partitions to disk.
User Memory (40%): Used to run user-defined functions (UDFs), store internal metadata tables, and handle JVM class loaders.

Code Examples

Python (PySpark) Configuration

python

from pyspark.sql import SparkSession

# Tune memory fraction ratios on session startup
spark = SparkSession.builder \
    .appName("MemoryManagement") \
    .config("spark.memory.fraction", "0.7") \
    .config("spark.memory.storageFraction", "0.3") \
    .getOrCreate()

df = spark.range(1, 1000)
df.collect()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

config(spark.memory.fraction

0.7)

range(1

1000)

collect()

Scala Configuration

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("MemoryScala")
  .config("spark.memory.fraction", "0.7")
  .config("spark.memory.storageFraction", "0.3")
  .getOrCreate()

val df = spark.range(1, 1000)
df.collect()