Tasks

Overview

Tasks are the smallest execution units in Apache Spark. Learn how Spark distributes tasks to executor cores to process data partitions in parallel.

What You Will Learn

In this lesson, you will learn:

Task Definition: The unit of execution mapped to a partition.
Executor Mapping: How cores process tasks concurrently.
Task Skew: Recognizing straggler tasks.

Detailed Concept Explanation

A Task represents a single thread of execution that runs the same code logic on a single partition of data.

text

Stage  ===>  Divided into Tasks (Task 1, Task 2, Task 3)  ===>  Distributed to Core Slots

Relationship between Tasks and Partitions

Spark creates exactly one task per partition in a stage. If a stage is processing a dataset with 200 partitions, Spark will launch 200 parallel tasks.
If an executor has 4 CPU cores, it can execute up to 4 tasks concurrently. The remaining tasks are queued until cores become free.

Code Examples

Python (PySpark) Implementation

python

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TasksConcept").getOrCreate()

# Create a dataset with 4 partitions (spawns 4 tasks for downstream actions)
df = spark.read.json("large_data.json")
df = df.repartition(4)

# Print partition count
print("Tasks to spawn on action:", df.rdd.getNumPartitions())

Expected Output

text

Tasks to spawn on action: 4

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

read.json

repartition(4)

getNumPartitions()

print()

Scala Implementation

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("TasksScala").getOrCreate()

val df = spark.read.json("large_data.json").repartition(4)
println(s"Tasks to spawn on action: ${df.rdd.getNumPartitions}")

Best Practices

Match tasks to core capacity: Ensure the number of partitions (tasks) is at least 2-4 times the total number of CPU cores in the cluster to prevent cores from sitting idle.