Stages

Overview

Learn how Spark divides a DAG into execution stages, and understand why shuffles create stage boundaries.

What You Will Learn

In this lesson, you will learn:

Stage Division: How Spark breaks jobs into stages.
Shuffle Boundary: Why wide transformations end stages.
Pipeling: Running multiple narrow transformations in a single stage.

Detailed Concept Explanation

When a Spark job executes, the DAG Scheduler groups transformations into distinct Stages.

How Stages are Divided

Stage Pipeling (Narrow Transformations): Transformations like map(), filter(), or select() do not require data to be reorganized across partitions. Spark joins these operations together, executing them in a single stage on the same thread.
Stage Barriers (Wide Transformations): Operations like groupBy(), join(), or distinct() require data with the same keys to be routed to the same partition. This network routing is called a Shuffle, which acts as a barrier that ends the current stage and starts a new one.

Code Examples

Python (PySpark) Implementation

python

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("StagesConcept").getOrCreate()

data1 = [("A", 10), ("B", 20)]
data2 = [("A", 30), ("C", 40)]
df1 = spark.createDataFrame(data1, ["key", "val1"])
df2 = spark.createDataFrame(data2, ["key", "val2"])

# Join triggers a shuffle, creating a stage boundary
joined = df1.join(df2, "key")
joined.show()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

createDF1

createDF2

join(key) [Shuffle Boundary / End Stage 1]

show()

Scala Implementation

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("StagesScala").getOrCreate()
import spark.implicits._

val df1 = Seq(("A", 10), ("B", 20)).toDF("key", "val1")
val df2 = Seq(("A", 30), ("C", 40)).toDF("key", "val2")

val joined = df1.join(df2, "key")
joined.show()

Interview Perspective

What triggers a new stage in a Spark job?

A new stage is triggered by a wide transformation (like a join, groupBy, or repartition) that requires a network shuffle. A shuffle reorganizes data partitions, creating a synchronization barrier where preceding stages must write their partition outputs before the succeeding stages can read and process them.