Task Scheduler

Overview

Learn how Spark's Task Scheduler distributes parallel tasks to executor cores and handles node failures.

What You Will Learn

Detailed Concept Explanation

While the DAG Scheduler handles high-level job compilation and stage division, the Task Scheduler is responsible for executing the individual tasks.

Core Responsibilities

• Task Launching: Receives a TaskSet from the DAG Scheduler, assigns tasks to executor nodes with free core slots, and starts execution.
• Locality-Aware Scheduling: Attempts to run tasks close to the target data partition (e.g. preferring the node that already holds the partition in cache or on local disk) to save network overhead.
• Error Handling: Monitors task execution. If a task fails, the Task Scheduler automatically retries it on another executor (up to 4 times by default, configured via spark.task.maxFailures). If all retries fail, the entire job fails.

Code Examples

Python (PySpark) Configuration

from pyspark.sql import SparkSession

# Tune task max failures count config
spark = SparkSession.builder \
    .appName("TaskScheduler") \
    .config("spark.task.maxFailures", "6") \
    .getOrCreate()

df = spark.range(1, 100)
print("Count:", df.count())

Execution Plan Diagram (Python & Scala)

Scala Configuration

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("TaskSchedulerScala")
  .config("spark.task.maxFailures", "6")
  .getOrCreate()

val df = spark.range(1, 100)
println(s"Count: ${df.count()}")

Related Topics

• DAG Scheduler
• Tasks
• Driver vs Executors