Hadoop YARN

Overview

Learn how Spark integrates with Hadoop YARN to manage resources, schedule jobs, and deploy in Client vs. Cluster modes.

What You Will Learn

In this lesson, you will learn:

YARN Architecture: ResourceManager, NodeManager, and ApplicationMaster.
Deploy Modes: Client mode vs. Cluster mode.
Resource Tuning: Allocating cores and memory for YARN containers.

Detailed Concept Explanation

Hadoop YARN (Yet Another Resource Negotiator) is a widely used distributed resource manager.

YARN Daemon Components

ResourceManager (RM): The cluster-level coordinator. Allocates resources across all applications.
NodeManager (NM): Runs on each node. Launches and monitors compute containers.
ApplicationMaster (AM): Created for each application to negotiate resources from RM and coordinate tasks.

Client vs. Cluster Deploy Modes

| Parameter | Client Mode | Cluster Mode | |-----------|-------------|--------------| | Driver Location | Runs inside the local submit client process. | Runs inside a YARN container on a worker node. | | Use Case | Interactive coding, testing, and debugging. | Production schedules (e.g. Airflow jobs). | | Resilience | If the client machine closes, the app crashes. | Highly resilient; YARN auto-restarts failed drivers. |

Code Examples

YARN Submit Command (Cluster Mode)

bash

# Submit job to YARN in cluster mode
spark-submit \
  --master yarn \
  --deploy-mode cluster \
  --num-executors 10 \
  --executor-cores 4 \
  --executor-memory 8G \
  production_job.py

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

spark-submit master(yarn) deploy-mode(cluster)

ResourceManager launches AM Container

AM launches Executors

Tasks execute on NodeManagers