DAG Scheduler

Overview

Learn about Spark's DAG Scheduler, the high-level scheduler that translates RDD lineage trees into stages of parallel tasks.

What You Will Learn

Detailed Concept Explanation

The DAG Scheduler is the high-level scheduling layer in Spark. It maps out the sequence of transformations (vertices) and the dependencies between them (edges).

Core Responsibilities

• Job Compilation: Triggered when an action is called.
• Stage Compilation: Divides the DAG into distinct stages. Pipelinable narrow transformations are grouped into the same stage, while wide transformations (which require network shuffles) act as boundaries that separate stages.
• TaskSet Creation: Compiles each stage into a TaskSet containing a task for each data partition, and submits it to the Task Scheduler.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("DAGScheduler").getOrCreate()

# Create lineage spanning a Join (2 stages)
df1 = spark.range(1, 100)
df2 = spark.range(1, 50)
joined = df1.join(df2, "id")

# Trigger job (DAG scheduler divides join into stages)
print("Count:", joined.count())

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("DAGSchedulerScala").getOrCreate()

val df1 = spark.range(1, 100)
val df2 = spark.range(1, 50)
val joined = df1.join(df2, "id")

println(s"Count: ${joined.count()}")

Related Topics

• Task Scheduler
• Stages
• DAG