Catalyst Optimizer

Overview

The Catalyst Optimizer is the heart of Spark SQL. Learn how it analyzes, optimizes, and compiles your queries into physical JVM execution plans.

What You Will Learn

Detailed Concept Explanation

The Catalyst Optimizer automatically optimizes Spark SQL and DataFrame queries. It compiles query representations into Java bytecode executed in JVM cores.

The Optimization Lifecycle

1. Analysis: Resolves table and column names against a schema catalog to produce an Analyzed Logical Plan.
2. Logical Optimization: Applies rule-based transformations to optimize the plan (e.g. constant folding, predicate pushdown, projection pruning) to produce an Optimized Logical Plan.
3. Physical Planning: Generates multiple physical plans and selects the most efficient plan based on Cost-Based Optimization (CBO) statistics (like table sizes or index keys).
4. Code Generation: Compiles the selected physical plan into Java bytecode at runtime (using Janino compiler), running as if written by hand.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Catalyst").getOrCreate()

data = [("A", 10), ("B", 20)]
df = spark.createDataFrame(data, ["key", "val"])

# Filter and select (Catalyst combines this so val is filtered first, and key is kept)
result = df.filter(df.val > 10).select("key")
result.explain(True)

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("CatalystScala").getOrCreate()
import spark.implicits._

val df = Seq(("A", 10), ("B", 20)).toDF("key", "val")
val result = df.filter($"val" > 10).select("key")

result.explain(true)

Interview Perspective

Related Topics

• Logical Plan
• Physical Plan
• Tungsten Engine