Persistence

Overview

Learn how to cache and persist RDDs in executor memory (RAM) or on disk to speed up iterative algorithms.

What You Will Learn

Detailed Concept Explanation

By default, Spark recomputes an RDD and all its lineage steps every time you trigger an Action.

To reuse an RDD across multiple actions without recomputing it, you can save it in executor memory or on disk. This is called Caching or Persistence.

API Methods

• rdd.cache(): A shortcut that saves the RDD with the default storage level: MEMORY_ONLY.
• rdd.persist(storageLevel): Allows you to choose from different storage levels, such as:
  • MEMORY_ONLY: Stores RDD as deserialized Java objects in the JVM. If it doesn't fit, some partitions are not cached and are recomputed on-the-fly.
  • MEMORY_AND_DISK: Stores partitions in memory, but spills excess partitions to disk if RAM is full.
  • DISK_ONLY: Stores RDD partitions entirely on disk.
  • MEMORY_ONLY_SER / MEMORY_AND_DISK_SER: Stores RDD as serialized (compact) byte arrays in memory, reducing memory footprint at the cost of higher CPU deserialization overhead.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession
from pyspark import StorageLevel

spark = SparkSession.builder.appName("Persistence").getOrCreate()
sc = spark.sparkContext
rdd = sc.parallelize([1, 2, 3, 4])

# Persist the RDD using MEMORY_AND_DISK level
rdd.persist(StorageLevel.MEMORY_AND_DISK)

# Trigger first action (computes and caches RDD)
print("Count:", rdd.count())

# Trigger second action (reads directly from cache)
print("Sum:", rdd.reduce(lambda x, y: x + y))

# Free up memory
rdd.unpersist()

Expected Output

Count: 4
Sum: 10

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession
import org.apache.spark.storage.StorageLevel

val spark = SparkSession.builder().appName("PersistenceScala").getOrCreate()
val sc = spark.sparkContext
val rdd = sc.parallelize(Seq(1, 2, 3, 4))

rdd.persist(StorageLevel.MEMORY_AND_DISK)

println(s"Count: ${rdd.count()}")
println(s"Sum: ${rdd.reduce(_ + _)}")

rdd.unpersist()

Expected Output

Count: 4
Sum: 10

Common Mistakes

• Forgetting to unpersist: Leaving cached datasets in memory after they are no longer needed. This wastes executor RAM, which can trigger garbage collection delays. Always call .unpersist() when you are done.

Best Practices

• Use serialized levels: When memory is tight, prefer serialized storage levels (MEMORY_ONLY_SER) to reduce memory usage, or use MEMORY_AND_DISK to prevent expensive recomputations of spilled partitions.

Interview Perspective

Interactive Challenges

Related Topics

• RDD Actions
• RDD vs DataFrame
• Cache vs Persist