intermediate
Cache vs Persist
Overview
Learn the differences between cache and persist to manage executor RAM and disk storage when caching datasets.
What You Will Learn
In this lesson, you will learn:
- Cache API: Default in-memory caching.
- Persist API: Custom storage configurations.
- Storage Levels: Choosing memory vs disk-backed cache layers.
Detailed Concept Explanation
By default, Spark re-evaluates query plans from scratch for every action. Caching allows you to save intermediate query results in executor memory/disk for reuse in downstream actions.
1. cache()
- Behavior: Shortcut that calls
persist()using the default storage level:MEMORY_AND_DISKfor DataFrames (andMEMORY_ONLYfor RDDs).
2. persist(storageLevel)
- Behavior: Allows you to specify different storage levels depending on your memory limits:
MEMORY_ONLY: Stores data in executor JVM memory as deserialized Java objects. If memory runs out, excess partitions are recomputed.MEMORY_ONLY_SER: Stores data as serialized byte arrays in memory. Takes less space but requires more CPU to decompress during reads.MEMORY_AND_DISK: Stores data in memory, but spills excess partitions to local disk if RAM is full.DISK_ONLY: Stores all data directly on local disk.
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
from pyspark import StorageLevel
spark = SparkSession.builder.appName("CachePersist").getOrCreate()
df = spark.range(1, 1000)
# Persist using serialized level to save memory
df.persist(StorageLevel.MEMORY_ONLY_SER)
# Trigger first action (computes and caches data)
print("Count:", df.count())
# Trigger second action (reads directly from cache)
print("Sum:", df.groupBy().sum().collect())
# Clean up memory when done
df.unpersist()
Expected Output
text
Count: 1000
Sum: [Row(sum(id)=499500)]
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
range(1
1000)
persist(MEMORY_ONLY_SER)
count()
sum() [Cached Read]
unpersist()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
import org.apache.spark.storage.StorageLevel
val spark = SparkSession.builder().appName("CachePersistScala").getOrCreate()
val df = spark.range(1, 1000)
df.persist(StorageLevel.MEMORY_ONLY_SER)
println(s"Count: ${df.count()}")
println(s"Sum: ${df.groupBy().sum().collect().mkString}")
df.unpersist()
Best Practices
- Always unpersist cached data: Leaving cached data in memory wastes executor RAM, which can trigger garbage collection pauses or OOM crashes in long-running jobs. Always call
.unpersist()when you are done.
Related Topics
DPA
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.