advanced

Logical Plan

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how Spark SQL builds the Logical Plan to compile your queries, tracking the journey from unresolved parse trees to optimized rule structures.

What You Will Learn

In this lesson, you will learn:
  • Unresolved Plan: The initial AST parsed from your query.
  • Analyzed Plan: Resolving table and column references against the Catalog.
  • Optimized Plan: Applying logical optimization rules (e.g. constant folding).

Detailed Concept Explanation

The Logical Plan is an abstract representation of the computational steps in your query, before any physical execution details (like memory limits or file formats) are decided.

It passes through three distinct phases inside the Catalyst Optimizer:

  1. Unresolved Logical Plan: Generated by parsing your SQL or DataFrame code. The syntax is checked, but column names and table names are not yet verified.
  2. Analyzed Logical Plan: Spark checks the columns and tables against the Catalog (schema metadata). If they exist, they are resolved.
  3. Optimized Logical Plan: Spark applies logical optimization rules (like pushing filters down, collapsing projections, and folding constants) to simplify the plan structure.

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("LogicalPlan").getOrCreate()

# Create a simple plan
df = spark.range(1, 100).filter(col("id") > 10).select("id")

# View only the logical phases of the query plan
df.explain(mode="extended")

Expected Output

text
== Parsed Logical Plan ==
...
== Analyzed Logical Plan ==
...
== Optimized Logical Plan ==
...

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
range(1
100)
filter(id > 10)
select(id)
explain(mode=extended)

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("LogicalPlanScala").getOrCreate()

val df = spark.range(1, 100).filter($"id" > 10).select("id")
df.explain(true)

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.