Hadoop YARN
Overview
Learn how Spark integrates with Hadoop YARN to manage resources, schedule jobs, and deploy in Client vs. Cluster modes.
What You Will Learn
In this lesson, you will learn:
- YARN Architecture: ResourceManager, NodeManager, and ApplicationMaster.
- Deploy Modes: Client mode vs. Cluster mode.
- Resource Tuning: Allocating cores and memory for YARN containers.
Detailed Concept Explanation
Hadoop YARN (Yet Another Resource Negotiator) is a widely used distributed resource manager.
YARN Daemon Components
- ResourceManager (RM): The cluster-level coordinator. Allocates resources across all applications.
- NodeManager (NM): Runs on each node. Launches and monitors compute containers.
- ApplicationMaster (AM): Created for each application to negotiate resources from RM and coordinate tasks.
Client vs. Cluster Deploy Modes
| Parameter | Client Mode | Cluster Mode | |-----------|-------------|--------------| | Driver Location | Runs inside the local submit client process. | Runs inside a YARN container on a worker node. | | Use Case | Interactive coding, testing, and debugging. | Production schedules (e.g. Airflow jobs). | | Resilience | If the client machine closes, the app crashes. | Highly resilient; YARN auto-restarts failed drivers. |
Code Examples
YARN Submit Command (Cluster Mode)
# Submit job to YARN in cluster mode
spark-submit \
--master yarn \
--deploy-mode cluster \
--num-executors 10 \
--executor-cores 4 \
--executor-memory 8G \
production_job.py
Execution Plan Diagram (Python & Scala)
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.