SPARK Reference Guide

Revision Time: 7 mins

Spark Executor Memory Architecture

Simple, intuitive breakdown of JVM Executor Memory structure: Reserved, User, Storage, Execution, and Off-Heap memory regions with configuration rules.

Spark Executor JVM Memory Architecture

--executor-memory 10g

Below is the visual structure of a single PySpark Executor JVM process. Spark automatically divides JVM memory into 4 primary regions managed by the Unified Memory Manager.

TOTAL EXECUTOR JVM HEAP (100% RAM)Usable Memory = Total Heap - 300MB

Reserved Memory300 MB

Hardcoded Spark internal tasks & stability.

Spark Memory Pool60% Usable RAM (`spark.memory.fraction=0.6`)

Execution Memory~30% Usable

Shuffles, Joins, Sorts, Aggregations.

Storage Memory~30% Usable

Cached DataFrames (`df.cache()`) & Broadcasts.

User Memory40% Usable RAM

Python UDFs, custom HashMaps, Metadata, and non-Spark data structures.

OFF-HEAP MEMORY (OPTIONAL DIRECT C-RAM)`spark.memory.offHeap.enabled=true` (Bypasses JVM GC)

Reserved Memory (300 MB)

Return: System Memory

Hardcoded system RAM reserved exclusively for Spark internal engine tasks and system stability.

Used In: System Overhead, Spark Internal Engine

Syntax signature:Fixed Constant = 300 MB

Code snippet:

python

If Executor RAM = 1024 MB:
Usable RAM = 1024 MB - 300 MB = 724 MB

Expected Output:Reserved: 300 MB (Static)

Remember: If `--executor-memory` is set below 460 MB, Spark job will immediately fail with `IllegalArgumentException: Executor memory must be at least 450 MB`.

Spark Memory Pool (60% of Usable RAM)

Return: Dynamic Pool

Shared dynamic memory pool used for Execution (Shuffles, Joins, Sorts) and Storage (Cached DataFrames, Broadcast Variables).

Used In: Shuffle Buffers, DataFrame Caching, Broadcast Lookups

Syntax signature:spark.memory.fraction = 0.6 (Default 60%)

Code snippet:

python

Spark Memory = (Total_Heap - 300MB) * 0.6

Expected Output:Dynamic shared region for Execution & Storage

Remember: Execution and Storage dynamically borrow memory from each other! If Storage is idle, Execution uses 100% of Spark Memory. If Storage needs space, it reclaims cached blocks up to storageFraction limit.

Execution Memory (Sub-pool of Spark Memory)

Return: Transient Memory

Used for intermediate data processing during Shuffles, Joins, Aggregations, Groupings, and Sorting operations.

Used In: SortMergeJoin, HashJoin, GroupBy Aggregations, Window Functions

Syntax signature:spark.memory.fraction * (1 - spark.memory.storageFraction)

Code snippet:

python

# During a 50GB Sort-Merge Join:
Execution Memory buffers intermediate partition buckets before writing to shuffle disk.

Expected Output:Short-lived intermediate join/sort buffers

Remember: Execution memory has eviction priority over Storage! If Execution needs RAM during a heavy join, it will forcibly evict cached blocks from Storage to disk.

Storage Memory (Sub-pool of Spark Memory)

Return: Persisted Memory

Stores cached DataFrames (`df.cache()`), persisted RDDs, and broadcast variables across executor nodes.

Used In: DataFrame Caching, RDD Persistence, Broadcast Variables

Syntax signature:spark.memory.storageFraction = 0.5 (Default 50% of Spark Memory)

Code snippet:

python

df.cache() # Stores serialized or un-serialized partition blocks in Storage Memory

Expected Output:Cached DataFrame Blocks & Broadcast Handles

Remember: Default immunity: 50% of Spark Memory is protected by `storageFraction`. Execution cannot evict cached blocks below this threshold.

User Memory (40% of Usable RAM)

Return: User Heap Memory

Stores user-defined data structures, custom Python dictionaries, UDF objects, RDD transformations (`map`, `flatMap`), and metadata.

Used In: Python UDFs, Custom Aggregators, Metadata, Internal RDD Maps

Syntax signature:User Memory = (Usable RAM) * (1 - spark.memory.fraction)

Code snippet:

python

# Custom Python UDF creating large dict:
my_dict = {i: str(i) for i in range(1000000)} # Resides in User Memory

Expected Output:Python/Java Objects & Custom Hash Maps

Common Mistakes:Creating 10GB in-memory lists inside PySpark UDFs, exhausting 40% User Memory.

Remember: If your custom Python functions build large in-memory lists or dicts, you will get OutOfMemory errors in User Memory. Optimize Python UDFs to process streams.

Off-Heap Memory (Tungsten Engine)

Return: Direct C Memory

Allocates native C-style memory outside the JVM Heap to completely eliminate Java Garbage Collection (GC) pauses.

Used In: Ultra Large Scale Clusters, Garbage Collection Elimination

Syntax signature:

spark.memory.offHeap.enabled = true
spark.memory.offHeap.size = 10g

Code snippet:

python

--conf spark.memory.offHeap.enabled=true --conf spark.memory.offHeap.size=8g

Expected Output:Direct native memory used for Tungsten binary rows

Remember: Off-Heap memory is managed directly in binary format by Project Tungsten, bypassing Java object serialization overhead.

Previous TopicOptimization & Performance Tuning Next TopicPySpark Interview Questions & Answers