Stages
Overview
Learn how Spark divides a DAG into execution stages, and understand why shuffles create stage boundaries.
What You Will Learn
In this lesson, you will learn:
- Stage Division: How Spark breaks jobs into stages.
- Shuffle Boundary: Why wide transformations end stages.
- Pipeling: Running multiple narrow transformations in a single stage.
Detailed Concept Explanation
When a Spark job executes, the DAG Scheduler groups transformations into distinct Stages.
How Stages are Divided
- Stage Pipeling (Narrow Transformations): Transformations like
map(),filter(), orselect()do not require data to be reorganized across partitions. Spark joins these operations together, executing them in a single stage on the same thread. - Stage Barriers (Wide Transformations): Operations like
groupBy(),join(), ordistinct()require data with the same keys to be routed to the same partition. This network routing is called a Shuffle, which acts as a barrier that ends the current stage and starts a new one.
Code Examples
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("StagesConcept").getOrCreate()
data1 = [("A", 10), ("B", 20)]
data2 = [("A", 30), ("C", 40)]
df1 = spark.createDataFrame(data1, ["key", "val1"])
df2 = spark.createDataFrame(data2, ["key", "val2"])
# Join triggers a shuffle, creating a stage boundary
joined = df1.join(df2, "key")
joined.show()
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("StagesScala").getOrCreate()
import spark.implicits._
val df1 = Seq(("A", 10), ("B", 20)).toDF("key", "val1")
val df2 = Seq(("A", 30), ("C", 40)).toDF("key", "val2")
val joined = df1.join(df2, "key")
joined.show()
Interview Perspective
What triggers a new stage in a Spark job?
A new stage is triggered by a wide transformation (like a join, groupBy, or repartition) that requires a network shuffle. A shuffle reorganizes data partitions, creating a synchronization barrier where preceding stages must write their partition outputs before the succeeding stages can read and process them.
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.