beginner

Tasks

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Tasks are the smallest execution units in Apache Spark. Learn how Spark distributes tasks to executor cores to process data partitions in parallel.

What You Will Learn

In this lesson, you will learn:
  • Task Definition: The unit of execution mapped to a partition.
  • Executor Mapping: How cores process tasks concurrently.
  • Task Skew: Recognizing straggler tasks.

Detailed Concept Explanation

A Task represents a single thread of execution that runs the same code logic on a single partition of data.

text
Stage  ===>  Divided into Tasks (Task 1, Task 2, Task 3)  ===>  Distributed to Core Slots

Relationship between Tasks and Partitions

  • Spark creates exactly one task per partition in a stage. If a stage is processing a dataset with 200 partitions, Spark will launch 200 parallel tasks.
  • If an executor has 4 CPU cores, it can execute up to 4 tasks concurrently. The remaining tasks are queued until cores become free.

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TasksConcept").getOrCreate()

# Create a dataset with 4 partitions (spawns 4 tasks for downstream actions)
df = spark.read.json("large_data.json")
df = df.repartition(4)

# Print partition count
print("Tasks to spawn on action:", df.rdd.getNumPartitions())

Expected Output

text
Tasks to spawn on action: 4

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
read.json
repartition(4)
getNumPartitions()
print()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("TasksScala").getOrCreate()

val df = spark.read.json("large_data.json").repartition(4)
println(s"Tasks to spawn on action: ${df.rdd.getNumPartitions}")

Best Practices

  • Match tasks to core capacity: Ensure the number of partitions (tasks) is at least 2-4 times the total number of CPU cores in the cluster to prevent cores from sitting idle.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.