beginner
Spark Core Architecture
8 min readLast updated: 2026-07-08
Overview
Understanding Spark's cluster managers, drivers, executors, and memory architectures is critical to writing optimized, leak-free distributed data pipelines.
What You Will Learn
In this lesson, you will learn:
- Master-Worker Topology: The coordinated roles of Driver Programs, Cluster Managers, and worker Executors.
- Executors & Slots: How executors process partition data concurrently inside designated CPU core slots.
- Partitions: How Spark divides large datasets into chunks to execute tasks concurrently.
- Driver vs Executor Roles: How to distinguish compilation and resource orchestration from task compute execution.
Detailed Concept Explanation
The Master-Worker Architecture
Apache Spark operates on a master-worker cluster architecture.
- Driver Node (Master): The coordinator of your application. It contains the SparkContext, evaluates user code, constructs Directed Acyclic Graphs (DAGs), schedules tasks, and maintains data lineage.
- Worker Node (Worker): Host machines in the cluster that run executor processes.
- Executor (Worker Process): A process launched on a worker node that executes individual tasks and stores cached partitions in memory.
- Cluster Manager: A central service that allocates container resources (like RAM and CPU) to the driver and worker executors. Supported managers include YARN (Hadoop), Kubernetes (containers), and Spark's native Standalone coordinator.
- Task: The smallest unit of execution sent by the driver to executor slots.
- Slot: A logical CPU core allocated inside an executor to process a single task partition in parallel.
- Partition: A logical chunk of your large dataset. Spark divides large files into smaller partitions so that different executor tasks can process them concurrently.
Architecture Diagram
Common Mistakes
- Hardcoding Cluster Masters: Configuring
.master("local[*]")inside production code locks the driver to run tasks on the local server instead of utilizing cluster executor containers. - Driver Memory Allocation Errors: Allocating excessive tasks to executors while leaving the driver memory small can trigger driver-side heap exhaustion due to returning partition summaries.
Performance Notes
- Master Setting: Always allow the cluster resource manager (YARN or Kubernetes) to dynamically allocate slot locations instead of hardcoding execution paths.
- Slot Balancing: Set 2-4 tasks per CPU core slot to keep executor cores fully utilized.
Best Practices
- Never Hardcode Master: Leave master settings out of Spark code; configure them via command-line arguments during
spark-submit. - Set Partition Sizes Properly: Aim for 100MB-200MB size per partition. Partitions too small cause massive task scheduling overhead; partitions too large trigger disk-spills and garbage collection delays.
Interview Perspective
Explain the difference between a Spark Driver and Executor.
When answering, focus on their distinct roles:
- The Driver: Runs the user's
main()function, initializes theSparkSession(andSparkContext), translates high-level code into physical logical steps (stages and tasks), and tracks execution states. - The Executor: Launched on worker nodes. Its sole responsibility is to receive tasks from the driver, execute them concurrently inside allocated slots, and cache designated partitions in memory. It reports status back to the driver.