Logical Plan

Overview

Learn how Spark SQL builds the Logical Plan to compile your queries, tracking the journey from unresolved parse trees to optimized rule structures.

What You Will Learn

In this lesson, you will learn:

Unresolved Plan: The initial AST parsed from your query.
Analyzed Plan: Resolving table and column references against the Catalog.
Optimized Plan: Applying logical optimization rules (e.g. constant folding).

Detailed Concept Explanation

The Logical Plan is an abstract representation of the computational steps in your query, before any physical execution details (like memory limits or file formats) are decided.

It passes through three distinct phases inside the Catalyst Optimizer:

Unresolved Logical Plan: Generated by parsing your SQL or DataFrame code. The syntax is checked, but column names and table names are not yet verified.
Analyzed Logical Plan: Spark checks the columns and tables against the Catalog (schema metadata). If they exist, they are resolved.
Optimized Logical Plan: Spark applies logical optimization rules (like pushing filters down, collapsing projections, and folding constants) to simplify the plan structure.

Code Examples

Python (PySpark) Implementation

python

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("LogicalPlan").getOrCreate()

# Create a simple plan
df = spark.range(1, 100).filter(col("id") > 10).select("id")

# View only the logical phases of the query plan
df.explain(mode="extended")

Expected Output

text

== Parsed Logical Plan ==
...
== Analyzed Logical Plan ==
...
== Optimized Logical Plan ==
...

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

range(1

100)

filter(id > 10)

select(id)

explain(mode=extended)

Scala Implementation

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("LogicalPlanScala").getOrCreate()

val df = spark.range(1, 100).filter($"id" > 10).select("id")
df.explain(true)