Logical Plan
Overview
Learn how Spark SQL builds the Logical Plan to compile your queries, tracking the journey from unresolved parse trees to optimized rule structures.
What You Will Learn
In this lesson, you will learn:
- Unresolved Plan: The initial AST parsed from your query.
- Analyzed Plan: Resolving table and column references against the Catalog.
- Optimized Plan: Applying logical optimization rules (e.g. constant folding).
Detailed Concept Explanation
The Logical Plan is an abstract representation of the computational steps in your query, before any physical execution details (like memory limits or file formats) are decided.
It passes through three distinct phases inside the Catalyst Optimizer:
- Unresolved Logical Plan: Generated by parsing your SQL or DataFrame code. The syntax is checked, but column names and table names are not yet verified.
- Analyzed Logical Plan: Spark checks the columns and tables against the Catalog (schema metadata). If they exist, they are resolved.
- Optimized Logical Plan: Spark applies logical optimization rules (like pushing filters down, collapsing projections, and folding constants) to simplify the plan structure.
Code Examples
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("LogicalPlan").getOrCreate()
# Create a simple plan
df = spark.range(1, 100).filter(col("id") > 10).select("id")
# View only the logical phases of the query plan
df.explain(mode="extended")
Expected Output
== Parsed Logical Plan ==
...
== Analyzed Logical Plan ==
...
== Optimized Logical Plan ==
...
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("LogicalPlanScala").getOrCreate()
val df = spark.range(1, 100).filter($"id" > 10).select("id")
df.explain(true)
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.