intermediate

Standalone Cluster

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how to configure, deploy, and manage a Spark Standalone Cluster using Spark's built-in resource manager.

What You Will Learn

In this lesson, you will learn:
  • Standalone Architecture: The Master and Worker daemon layout.
  • Configuration scripts: Launching clusters using helper shells.
  • Job Submissions: Deploying applications using master urls.

Detailed Concept Explanation

A Standalone Cluster is a simple, built-in cluster manager provided with Spark. If you do not have access to large infrastructure setups like YARN or Kubernetes, you can launch a standalone cluster yourself.

Daemon Components

  1. Master Daemon: Runs on one node in the cluster. It manages resources, monitors worker health, and coordinates driver applications.
  2. Worker Daemon: Runs on each worker node. It monitors resource allocations (CPU cores, RAM) and starts executor processes when directed by the master.

Control Scripts

Spark provides helper scripts inside its sbin/ directory to control standalone daemons:

  • ./sbin/start-master.sh: Starts the Master daemon.
  • ./sbin/start-worker.sh <master-url>: Starts a Worker daemon.
  • ./sbin/start-all.sh: Launches Master and Workers on all nodes listed in the configuration file.

Code Examples

Standalone Submit Command

bash
# Submit job to Standalone master URL
spark-submit \
  --master spark://master-ip:7077 \
  --deploy-mode client \
  --executor-memory 2G \
  --executor-cores 2 \
  my_app.py

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
Master Web UI (7077)
Worker Daemon registers
spark-submit master(spark://master:7077)
executors started
tasks executed

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.