Tungsten Engine

Overview

Tungsten is Spark's physical execution optimizer. Learn how it bypasses Java object overheads, manages off-heap memory, and speeds up computation.

What You Will Learn

Detailed Concept Explanation

While Catalyst focuses on logical optimization, Tungsten optimizes physical CPU and memory layouts.

Key Optimization Pillars

1. Binary Row Format: Java objects carry massive memory overheads (e.g. a simple 4-byte integer string can take up to 24 bytes of JVM metadata). Tungsten stores rows as compact, raw binary bytes, reducing memory usage and saving garbage collection cycles.
2. Off-Heap Memory Management: Bypasses the standard JVM garbage collector by managing allocations directly in off-heap memory arrays.
3. Whole-Stage Code Generation: Collapses multiple physical plans (e.g. scanning, filtering, and projecting) into a single, clean Java function, eliminating virtual method calls and keeping data in CPU registers.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Tungsten").getOrCreate()

# A query that Catalyst/Tungsten collapses into a single function
df = spark.range(1, 1000000)
result = df.filter(df.id % 2 == 0).selectExpr("id * 2")

# Codegen is represented in explain plan by star symbol (*)
result.explain()

Expected Output

== Physical Plan ==
*(1) Project [(id#0L * 2) AS (id * 2)#2L]
+- *(1) Filter ((id#0L % 2) = 0)
   +- *(1) Range (1, 1000000, step=1, splits=2)

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("TungstenScala").getOrCreate()

val df = spark.range(1, 1000000)
val result = df.filter($"id" % 2 === 0).selectExpr("id * 2")

result.explain()

Interview Perspective

Related Topics

• Memory Management
• Catalyst Optimizer
• Whole Stage Code Generation