Physical Plan

Overview

Learn how Spark translates the optimized logical plan into a Physical Plan mapping physical execution algorithms like joins and scans.

What You Will Learn

In this lesson, you will learn:

Physical Planning: Translating logical steps into physical algorithms.
Cost-Based Optimizer (CBO): Selecting the most efficient plan.
Code Generation: Translating steps into runnable JVM bytecode.

Detailed Concept Explanation

While the Logical Plan defines what calculations to run, the Physical Plan defines how to execute them on the cluster hardware.

Physical Planning Stages

Generation of Physical Plans: Spark generates multiple physical strategies (e.g. implementing a join as a Sort-Merge Join vs a Broadcast Hash Join).
Cost-Based Optimization (CBO): Spark evaluates the statistics (like table sizes) of each generated physical plan and estimates their cost, selecting the plan with the lowest computational cost.
Execution: Compiles the selected plan into Java bytecode at runtime using Whole-Stage Code Generation.

Code Examples

Python (PySpark) Implementation

python

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PhysicalPlan").getOrCreate()

# Simple join query
df1 = spark.range(1, 100)
df2 = spark.range(1, 50)
joined = df1.join(df2, "id")

# Print the compiled physical plan
joined.explain()

Expected Output

text

== Physical Plan ==
AdaptiveSparkPlan isAdaptive=true
+- BroadcastHashJoin [id#0L], [id#2L], Inner, BuildRight
...

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

rangeDF1

rangeDF2

join(id)

explain()

Scala Implementation

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("PhysicalPlanScala").getOrCreate()

val df1 = spark.range(1, 100)
val df2 = spark.range(1, 50)
val joined = df1.join(df2, "id")

joined.explain()