Driver vs Executors

Overview

Every Apache Spark application runs as a coordinated set of processes: one Driver process and multiple Executor processes. Understanding the division of labor between these two is critical for debugging resource issues.

What You Will Learn

Detailed Concept Explanation

The Spark Architecture Split

When you run a Spark application, Spark launches a master process (the Driver) and several worker processes (Executors) across your cluster.

               +-------------------+
               |    Driver Node    | (Main Orchestrator)
               |   [SparkContext]  |
               +-------------------+
                  /             \
                 /               \  (Direct Task Scheduling)
                v                 v
        +---------------+   +---------------+
        |  Executor 1   |   |  Executor 2   | (Compute & Storage Workers)
        | [Task] [Task] |   | [Task] [Task] |
        +---------------+   +---------------+

The Driver Node

The Driver is the "brain" of your program. It is the machine where your main() method runs. Its primary responsibilities are:

1. Orchestrating execution: It translates your Python, Scala, or SQL code into logical and physical execution plans.
2. Scheduling tasks: It breaks plans down into stages and schedules individual tasks across the executors.
3. Tracking state: It monitors executor health and coordinates recovery if a node fails.
4. Returning results: When you call an action like .collect(), the data is sent back to the driver.

The Executor Nodes

Executors are the "muscles" of your program. They run on the worker nodes in the cluster. Their primary responsibilities are:

1. Running tasks: They execute the individual task threads assigned to them by the driver.
2. Caching partitions: They store partition data in memory (RAM) or on local disk when you call .cache().
3. Reporting status: They send task results and executor health heartbeat metrics back to the driver.

Common Mistakes

• Triggering OOM on the Driver: Calling .collect() on a huge dataset pulls all partition records from the executors into the driver's memory. If the dataset size exceeds the driver's memory, the application crashes with an Out Of Memory (OOM) error.
• Heavy Local Loops: Writing custom loops that execute locally on the driver machine instead of distributed transformations on the executors.

Performance Notes

• Executor Allocations: Balance the size of your executors. Launching executors that are too large (e.g. 64 CPU cores each) causes high garbage collection delays. Launching executors that are too small (e.g. 1 core each) wastes memory overhead.

Best Practices

• Limit Data Collection: Use .take(n) or write data directly to a storage sink instead of calling .collect() on large datasets.
• Match Cores to Slots: Standard practice is to allocate between 2 and 5 CPU core slots per executor to maximize processing efficiency.

Interview Perspective

Interactive Challenges

Related Topics

• Spark Core Architecture
• Cluster Managers
• Spark Application Lifecycle