advanced

Hadoop YARN

By DataPlayArena Technical Review BoardUpdated 2026-07-0910 min readRuntime Verified

Overview

Learn how Spark integrates with Hadoop YARN to manage resources, schedule jobs, and deploy in Client vs. Cluster modes.

What You Will Learn

In this lesson, you will learn:
  • YARN Architecture: ResourceManager, NodeManager, and ApplicationMaster.
  • Deploy Modes: Client mode vs. Cluster mode.
  • Resource Tuning: Allocating cores and memory for YARN containers.

Detailed Concept Explanation

Hadoop YARN (Yet Another Resource Negotiator) is a widely used distributed resource manager.

YARN Daemon Components

  • ResourceManager (RM): The cluster-level coordinator. Allocates resources across all applications.
  • NodeManager (NM): Runs on each node. Launches and monitors compute containers.
  • ApplicationMaster (AM): Created for each application to negotiate resources from RM and coordinate tasks.

Client vs. Cluster Deploy Modes

| Parameter | Client Mode | Cluster Mode | |-----------|-------------|--------------| | Driver Location | Runs inside the local submit client process. | Runs inside a YARN container on a worker node. | | Use Case | Interactive coding, testing, and debugging. | Production schedules (e.g. Airflow jobs). | | Resilience | If the client machine closes, the app crashes. | Highly resilient; YARN auto-restarts failed drivers. |


Code Examples

YARN Submit Command (Cluster Mode)

bash
# Submit job to YARN in cluster mode
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --num-executors 10 \
  --executor-cores 4 \
  --executor-memory 8G \
  production_job.py

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
spark-submit master(yarn) deploy-mode(cluster)
ResourceManager launches AM Container
AM launches Executors
Tasks execute on NodeManagers

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.