advanced

Task Scheduler

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how Spark's Task Scheduler distributes parallel tasks to executor cores and handles node failures.

What You Will Learn

In this lesson, you will learn:
  • Task Distribution: Launching tasks on executors.
  • Locality-Aware Scheduling: Running tasks close to data partitions.
  • Task Retries: Handling task failures and retries automatically.

Detailed Concept Explanation

While the DAG Scheduler handles high-level job compilation and stage division, the Task Scheduler is responsible for executing the individual tasks.

Core Responsibilities

  • Task Launching: Receives a TaskSet from the DAG Scheduler, assigns tasks to executor nodes with free core slots, and starts execution.
  • Locality-Aware Scheduling: Attempts to run tasks close to the target data partition (e.g. preferring the node that already holds the partition in cache or on local disk) to save network overhead.
  • Error Handling: Monitors task execution. If a task fails, the Task Scheduler automatically retries it on another executor (up to 4 times by default, configured via spark.task.maxFailures). If all retries fail, the entire job fails.

Code Examples

Python (PySpark) Configuration

python
from pyspark.sql import SparkSession

# Tune task max failures count config
spark = SparkSession.builder \
    .appName("TaskScheduler") \
    .config("spark.task.maxFailures", "6") \
    .getOrCreate()

df = spark.range(1, 100)
print("Count:", df.count())

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
config(spark.task.maxFailures
6)
range
count()
print()

Scala Configuration

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("TaskSchedulerScala")
  .config("spark.task.maxFailures", "6")
  .getOrCreate()

val df = spark.range(1, 100)
println(s"Count: ${df.count()}")

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.