Job Execution Flow

Overview

Learn how Apache Spark translates your code into a physical running application, following the journey from Driver compilation to parallel executor tasks.

What You Will Learn

In this lesson, you will learn:

The Lifecycle Journey: How a job starts, divides, and executes.
DAG & Task Schedulers: The role of Spark's internal schedulers.
Worker Tasks: How executors run code blocks in parallel.

Detailed Concept Explanation

When you submit a Spark program, Spark does not execute it line-by-line in a simple sequence. It builds an optimized execution pipeline.

Here is the step-by-step lifecycle flow of a Spark Job:

text

User Code (Action called) 
   │
   ▼
1. Driver Analyzes Code (Catalyst constructs DAG)
   │
   ▼
2. DAG Scheduler (Divides DAG into Stages at shuffle barriers)
   │
   ▼
3. Task Scheduler (Launches Tasks for partitions to Executors)
   │
   ▼
4. Executors (Worker threads run task bytecode on CPU cores)
   │
   ▼
5. Driver (Collects final results or writes to storage sink)

1. Job Trigger

Every time you call an Action (like collect() or save()), a Job is triggered. A single Spark application can run many jobs.

2. DAG compilation

The Driver's DAG Scheduler compiles the lineage of transformations into a Directed Acyclic Graph (DAG) plan.

3. Stage division

The DAG Scheduler analyzes shuffle boundaries:

Pipelines narrow dependencies together into a single stage.
Sets shuffle boundaries (wide dependencies) to end the current stage and start the next.

4. Task Scheduling

The DAG Scheduler passes the stages to the Task Scheduler. The Task Scheduler creates a set of Tasks (one per partition) and distributes them to executor worker nodes based on data locality (preferring workers that already store the partition data).

Code Examples

Python (PySpark) Implementation

python

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("JobExecutionFlow").getOrCreate()

# 1. Read data (Lazy: builds plan structure)
df = spark.range(1, 1000)

# 2. Filter (Lazy: appends transformation to DAG)
filtered_df = df.filter(df.id % 2 == 0)

# 3. Action called (Triggers the Job execution flow!)
print("Total rows:", filtered_df.count())

Expected Output

text

Total rows: 499

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

range(1

1000)

filter(id % 2 == 0) [DAG built]

count() [Action triggers Job Execution Flow]

print()

Scala Implementation

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("JobFlowScala").getOrCreate()

val df = spark.range(1, 1000)
val filtered = df.filter($"id" % 2 === 0)

// Action triggers execution flow
println(s"Total rows: ${filtered.count()}")

Best Practices

Observe Web UI Stages: Open the Spark Web UI (port 4040 by default) during execution. It visualizes the compiled DAG, stages, and tasks in real-time, helping you identify bottlenecks.

Interview Perspective

What is the difference between a Job, a Stage, and a Task in Spark?

Job: The high-level execution pipeline triggered whenever an Action (e.g. collect(), count()) is called.
Stage: A subdivision of a Job. Stages are separated by wide transformations (shuffles). Narrow transformations are pipelined and run in the same stage.
Task: The smallest unit of execution. A Task runs a stage's code on a single partition of data. Spark launches one task per partition.

Interactive Challenges

Challenge 1: Identify job triggers (Beginner)

Which of these Spark operations will trigger a new Job execution flow: select(), filter(), or show()?