Lazy Evaluation

Overview

Lazy Evaluation is a fundamental core performance strategy in Apache Spark. Learn how Spark delays execution until an action is called, allowing query optimizations.

What You Will Learn

Detailed Concept Explanation

In Spark, when you write transformations (like map(), filter(), or select()), Spark does not execute them immediately. Instead, it records these operations as a series of instructions on how to build the final dataset.

Execution only happens when you trigger an Action (like collect(), show(), or count()).

User Code: read() -> filter() -> select() ---> count() (Action)
Execution:  [-------- Lazy Planning --------] ---> [ Spark Runs Code! ]

Why Lazy Evaluation?

• Optimizations: By knowing the entire chain of transformations before execution, Spark's Catalyst Optimizer can combine operations, prune columns, and skip unnecessary data reads.
• Fault Tolerance: If a partition is lost, Spark re-runs the recorded lineage graph to recompute only the lost data.

Code Examples

Input Dataset Preview

Below is the users transaction dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("LazyEvaluation").getOrCreate()

# Read data (Lazy: no job runs yet)
data = [("Alice", 25), ("Bob", 17)]
df = spark.createDataFrame(data, ["name", "age"])

# Apply filter (Lazy: only records metadata instructions)
filtered_df = df.filter(df.age >= 18)

# Trigger Action (Spark compiles and executes the plan here)
filtered_df.show()

Expected Output

+-----+---+
| name|age|
+-----+---+
|Alice| 25|
+-----+---+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("LazyEvaluationScala").getOrCreate()
import spark.implicits._

val df = Seq(("Alice", 25), ("Bob", 17)).toDF("name", "age")
val filtered = df.filter($"age" >= 18)

// Action triggers execution
filtered.show()

Expected Output

+-----+---+
| name|age|
+-----+---+
|Alice| 25|
+-----+---+

Best Practices

• Never trigger unnecessary actions: Avoid calling .count() or .show() in the middle of production pipelines just for debugging, as it forces Spark to execute the plan early, hurting performance.

Related Topics

• DAG
• Introduction to DataFrames
• Catalyst Optimizer