Standalone Cluster
Overview
Learn how to configure, deploy, and manage a Spark Standalone Cluster using Spark's built-in resource manager.
What You Will Learn
In this lesson, you will learn:
- Standalone Architecture: The Master and Worker daemon layout.
- Configuration scripts: Launching clusters using helper shells.
- Job Submissions: Deploying applications using master urls.
Detailed Concept Explanation
A Standalone Cluster is a simple, built-in cluster manager provided with Spark. If you do not have access to large infrastructure setups like YARN or Kubernetes, you can launch a standalone cluster yourself.
Daemon Components
- Master Daemon: Runs on one node in the cluster. It manages resources, monitors worker health, and coordinates driver applications.
- Worker Daemon: Runs on each worker node. It monitors resource allocations (CPU cores, RAM) and starts executor processes when directed by the master.
Control Scripts
Spark provides helper scripts inside its sbin/ directory to control standalone daemons:
./sbin/start-master.sh: Starts the Master daemon../sbin/start-worker.sh <master-url>: Starts a Worker daemon../sbin/start-all.sh: Launches Master and Workers on all nodes listed in the configuration file.
Code Examples
Standalone Submit Command
# Submit job to Standalone master URL
spark-submit \
--master spark://master-ip:7077 \
--deploy-mode client \
--executor-memory 2G \
--executor-cores 2 \
my_app.py
Execution Plan Diagram (Python & Scala)
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.