intermediate

Stages

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how Spark divides a DAG into execution stages, and understand why shuffles create stage boundaries.

What You Will Learn

In this lesson, you will learn:
  • Stage Division: How Spark breaks jobs into stages.
  • Shuffle Boundary: Why wide transformations end stages.
  • Pipeling: Running multiple narrow transformations in a single stage.

Detailed Concept Explanation

When a Spark job executes, the DAG Scheduler groups transformations into distinct Stages.

How Stages are Divided

  • Stage Pipeling (Narrow Transformations): Transformations like map(), filter(), or select() do not require data to be reorganized across partitions. Spark joins these operations together, executing them in a single stage on the same thread.
  • Stage Barriers (Wide Transformations): Operations like groupBy(), join(), or distinct() require data with the same keys to be routed to the same partition. This network routing is called a Shuffle, which acts as a barrier that ends the current stage and starts a new one.

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("StagesConcept").getOrCreate()

data1 = [("A", 10), ("B", 20)]
data2 = [("A", 30), ("C", 40)]
df1 = spark.createDataFrame(data1, ["key", "val1"])
df2 = spark.createDataFrame(data2, ["key", "val2"])

# Join triggers a shuffle, creating a stage boundary
joined = df1.join(df2, "key")
joined.show()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
createDF1
createDF2
join(key) [Shuffle Boundary / End Stage 1]
show()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("StagesScala").getOrCreate()
import spark.implicits._

val df1 = Seq(("A", 10), ("B", 20)).toDF("key", "val1")
val df2 = Seq(("A", 30), ("C", 40)).toDF("key", "val2")

val joined = df1.join(df2, "key")
joined.show()

Interview Perspective

What triggers a new stage in a Spark job?

A new stage is triggered by a wide transformation (like a join, groupBy, or repartition) that requires a network shuffle. A shuffle reorganizes data partitions, creating a synchronization barrier where preceding stages must write their partition outputs before the succeeding stages can read and process them.


Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.