Explain Plan

Overview

Learn how to use the explain() method to read, analyze, and troubleshoot Spark query execution plans.

What You Will Learn

Detailed Concept Explanation

To optimize slow queries, you must understand what Spark is doing under the hood. The .explain() method outputs a detailed text summary of Spark's execution plans.

Explain Modes (Spark 3+)

• explain(mode="simple"): Shows only the physical plan.
• explain(mode="extended"): Shows all plan phases (Parsed, Analyzed, Optimized, Physical).
• explain(mode="codegen"): Shows the compiled Java bytecode generated for execution.
• explain(mode="cost"): Displays join/aggregate costs and size statistics.
• explain(mode="formatted"): Displays a simplified layout mapping node trees.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExplainPlan").getOrCreate()

df = spark.range(1, 100).filter(col("id") > 10)

# Print a formatted execution report
df.explain(mode="formatted")

Expected Output

== Physical Plan ==
* Filter (id#0L > 10)
  +- * Range (1, 100, step=1, splits=2)

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ExplainScala").getOrCreate()

val df = spark.range(1, 100).filter($"id" > 10)
df.explain("formatted")

Best Practices

• Check for PushedFilters: In the physical plan print, look for PushedFilters. This indicates that Spark is successfully pushing filters down to the file storage layer.

Related Topics

• Logical Plan
• Physical Plan
• Catalyst Optimizer