Cache vs Persist

Overview

Learn the differences between cache and persist to manage executor RAM and disk storage when caching datasets.

What You Will Learn

Detailed Concept Explanation

By default, Spark re-evaluates query plans from scratch for every action. Caching allows you to save intermediate query results in executor memory/disk for reuse in downstream actions.

1. cache()

• Behavior: Shortcut that calls persist() using the default storage level: MEMORY_AND_DISK for DataFrames (and MEMORY_ONLY for RDDs).

2. persist(storageLevel)

• Behavior: Allows you to specify different storage levels depending on your memory limits:
  • MEMORY_ONLY: Stores data in executor JVM memory as deserialized Java objects. If memory runs out, excess partitions are recomputed.
  • MEMORY_ONLY_SER: Stores data as serialized byte arrays in memory. Takes less space but requires more CPU to decompress during reads.
  • MEMORY_AND_DISK: Stores data in memory, but spills excess partitions to local disk if RAM is full.
  • DISK_ONLY: Stores all data directly on local disk.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession
from pyspark import StorageLevel

spark = SparkSession.builder.appName("CachePersist").getOrCreate()

df = spark.range(1, 1000)

# Persist using serialized level to save memory
df.persist(StorageLevel.MEMORY_ONLY_SER)

# Trigger first action (computes and caches data)
print("Count:", df.count())

# Trigger second action (reads directly from cache)
print("Sum:", df.groupBy().sum().collect())

# Clean up memory when done
df.unpersist()

Expected Output

Count: 1000
Sum: [Row(sum(id)=499500)]

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession
import org.apache.spark.storage.StorageLevel

val spark = SparkSession.builder().appName("CachePersistScala").getOrCreate()

val df = spark.range(1, 1000)
df.persist(StorageLevel.MEMORY_ONLY_SER)

println(s"Count: ${df.count()}")
println(s"Sum: ${df.groupBy().sum().collect().mkString}")

df.unpersist()

Best Practices

• Always unpersist cached data: Leaving cached data in memory wastes executor RAM, which can trigger garbage collection pauses or OOM crashes in long-running jobs. Always call .unpersist() when you are done.

Related Topics

• Persistence
• Memory Management
• RDD vs DataFrame