Standalone Cluster

Overview

Learn how to configure, deploy, and manage a Spark Standalone Cluster using Spark's built-in resource manager.

What You Will Learn

In this lesson, you will learn:

Standalone Architecture: The Master and Worker daemon layout.
Configuration scripts: Launching clusters using helper shells.
Job Submissions: Deploying applications using master urls.

Detailed Concept Explanation

A Standalone Cluster is a simple, built-in cluster manager provided with Spark. If you do not have access to large infrastructure setups like YARN or Kubernetes, you can launch a standalone cluster yourself.

Daemon Components

Master Daemon: Runs on one node in the cluster. It manages resources, monitors worker health, and coordinates driver applications.
Worker Daemon: Runs on each worker node. It monitors resource allocations (CPU cores, RAM) and starts executor processes when directed by the master.

Control Scripts

Spark provides helper scripts inside its sbin/ directory to control standalone daemons:

./sbin/start-master.sh: Starts the Master daemon.
./sbin/start-worker.sh <master-url>: Starts a Worker daemon.
./sbin/start-all.sh: Launches Master and Workers on all nodes listed in the configuration file.

Code Examples

Standalone Submit Command

bash

# Submit job to Standalone master URL
spark-submit \
  --master spark://master-ip:7077 \
  --deploy-mode client \
  --executor-memory 2G \
  --executor-cores 2 \
  my_app.py

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

Master Web UI (7077)

Worker Daemon registers

spark-submit master(spark://master:7077)

executors started

tasks executed