intermediate
Explain Plan
Overview
Learn how to use the explain() method to read, analyze, and troubleshoot Spark query execution plans.
What You Will Learn
In this lesson, you will learn:
- Explain API: Querying execution details on DataFrames.
- Explain Modes: Extended, formatted, and cost modes.
- Troubleshooting: Identifying shuffles and filters in plan dumps.
Detailed Concept Explanation
To optimize slow queries, you must understand what Spark is doing under the hood. The .explain() method outputs a detailed text summary of Spark's execution plans.
Explain Modes (Spark 3+)
explain(mode="simple"): Shows only the physical plan.explain(mode="extended"): Shows all plan phases (Parsed, Analyzed, Optimized, Physical).explain(mode="codegen"): Shows the compiled Java bytecode generated for execution.explain(mode="cost"): Displays join/aggregate costs and size statistics.explain(mode="formatted"): Displays a simplified layout mapping node trees.
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("ExplainPlan").getOrCreate()
df = spark.range(1, 100).filter(col("id") > 10)
# Print a formatted execution report
df.explain(mode="formatted")
Expected Output
text
== Physical Plan ==
* Filter (id#0L > 10)
+- * Range (1, 100, step=1, splits=2)
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
range
filter(id > 10)
explain(mode=formatted)
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("ExplainScala").getOrCreate()
val df = spark.range(1, 100).filter($"id" > 10)
df.explain("formatted")
Best Practices
- Check for PushedFilters: In the physical plan print, look for
PushedFilters. This indicates that Spark is successfully pushing filters down to the file storage layer.
Related Topics
DPA
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.