beginner
Spark Application Lifecycle
10 min readLast updated: 2026-07-08
Overview
Understanding the lifecycle of a Spark application helps you trace exactly what happens behind the scenes from the moment you submit your code until the job finishes.
What You Will Learn
In this lesson, you will learn:
- Launch Phase: How your driver process is started and executor resources are requested.
- Compilation Phase: How Spark translates code commands into stages and tasks.
- Execution Phase: How tasks are sent to executor slots and processed.
- Termination: How sessions release resources back to the cluster.
Detailed Concept Explanation
Stage 1: Application Submission
You submit your Spark application using the spark-submit command-line utility.
- The driver process starts (either on your local machine or in a container on the cluster).
- The driver initializes a SparkSession (which starts the SparkContext under the hood).
Stage 2: Resource Allocation
- The driver contacts the Cluster Manager (e.g. YARN, K8s).
- The Cluster Manager identifies worker nodes and launches Executor processes with the CPU and memory resources you requested.
Stage 3: Task Compilation
- As your code defines transformations, the driver's Catalyst Optimizer builds a Directed Acyclic Graph (DAG) of logical plans.
- When your code runs an Action (such as
.show()or.count()), the driver compiles the DAG into Stages (separated by shuffle boundaries). - The driver breaks stages down into individual Tasks (one task per data partition).
Stage 4: Task Execution
- The driver schedules and sends tasks directly to the executor nodes.
- Executors run the tasks in parallel inside their available CPU core slots.
- Executors report task status and results back to the driver.
Stage 5: Termination
- Once the program finishes, your code calls
spark.stop(). - The driver terminates, and the Cluster Manager shuts down the executors, releasing all resources back to the cluster.
Common Mistakes
- Forgetting to Stop SparkSession: Leaving a Spark application running without calling
spark.stop()keeps executor resources reserved, blocking other users from submitting jobs to the cluster.
Best Practices
- Always Clean Up: Always close your session using
spark.stop()in your code, or run it inside a try-finally block.
Interview Perspective
Describe the lifecycle steps of a Spark application when an Action is triggered.
When an action is triggered:
- The Driver intercepts the call and compiles the logical DAG plan into physical stages (divided by network shuffle boundaries).
- The stages are broken into individual tasks (matching the partitions count).
- The Driver schedules and assigns tasks to the executors over the network.
- Executors process tasks in parallel slots and return execution status updates to the driver.