advanced
Task Scheduler
Overview
Learn how Spark's Task Scheduler distributes parallel tasks to executor cores and handles node failures.
What You Will Learn
In this lesson, you will learn:
- Task Distribution: Launching tasks on executors.
- Locality-Aware Scheduling: Running tasks close to data partitions.
- Task Retries: Handling task failures and retries automatically.
Detailed Concept Explanation
While the DAG Scheduler handles high-level job compilation and stage division, the Task Scheduler is responsible for executing the individual tasks.
Core Responsibilities
- Task Launching: Receives a
TaskSetfrom the DAG Scheduler, assigns tasks to executor nodes with free core slots, and starts execution. - Locality-Aware Scheduling: Attempts to run tasks close to the target data partition (e.g. preferring the node that already holds the partition in cache or on local disk) to save network overhead.
- Error Handling: Monitors task execution. If a task fails, the Task Scheduler automatically retries it on another executor (up to 4 times by default, configured via
spark.task.maxFailures). If all retries fail, the entire job fails.
Code Examples
Python (PySpark) Configuration
python
from pyspark.sql import SparkSession
# Tune task max failures count config
spark = SparkSession.builder \
.appName("TaskScheduler") \
.config("spark.task.maxFailures", "6") \
.getOrCreate()
df = spark.range(1, 100)
print("Count:", df.count())
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
config(spark.task.maxFailures
6)
range
count()
print()
Scala Configuration
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.appName("TaskSchedulerScala")
.config("spark.task.maxFailures", "6")
.getOrCreate()
val df = spark.range(1, 100)
println(s"Count: ${df.count()}")
Related Topics
DPA
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.