advanced
DAG Scheduler
Overview
Learn about Spark's DAG Scheduler, the high-level scheduler that translates RDD lineage trees into stages of parallel tasks.
What You Will Learn
In this lesson, you will learn:
- Lineage Trees: Mapping dependencies between operations.
- Stage Division: Dividing graphs at shuffle barriers.
- TaskSet Submission: Handing task batches to the Task Scheduler.
Detailed Concept Explanation
The DAG Scheduler is the high-level scheduling layer in Spark. It maps out the sequence of transformations (vertices) and the dependencies between them (edges).
Core Responsibilities
- Job Compilation: Triggered when an action is called.
- Stage Compilation: Divides the DAG into distinct stages. Pipelinable narrow transformations are grouped into the same stage, while wide transformations (which require network shuffles) act as boundaries that separate stages.
- TaskSet Creation: Compiles each stage into a
TaskSetcontaining a task for each data partition, and submits it to the Task Scheduler.
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("DAGScheduler").getOrCreate()
# Create lineage spanning a Join (2 stages)
df1 = spark.range(1, 100)
df2 = spark.range(1, 50)
joined = df1.join(df2, "id")
# Trigger job (DAG scheduler divides join into stages)
print("Count:", joined.count())
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
rangeDF1
rangeDF2
join(id) [DAG divides stages]
count() [Submit TaskSets]
print()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("DAGSchedulerScala").getOrCreate()
val df1 = spark.range(1, 100)
val df2 = spark.range(1, 50)
val joined = df1.join(df2, "id")
println(s"Count: ${joined.count()}")
Related Topics
DPA
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.