Local Mode
Overview
Learn how to run Spark in Local Mode on a single machine for fast development, code testing, and prototyping.
What You Will Learn
In this lesson, you will learn:
- Local Mode Architecture: Driver and executor executing in a single local JVM.
- Master String: Configuring threads using
local[N]master configurations. - Development Cycles: Fast local testing cycles.
Detailed Concept Explanation
Local Mode is the simplest deployment mode in Apache Spark. Instead of launching processes across a distributed network of nodes, Spark runs the Driver, executors, and master threads inside a single JVM on your local machine.
Master Configuration Values
You configure the local thread count by setting the master parameter:
local: Runs Spark on a single thread. (Slowest, no parallel execution).local[4]: Runs Spark using exactly 4 execution threads.local[*]: Runs Spark using as many threads as there are logical CPU cores on your machine.
When to use it
- Code testing and local prototyping.
- Running unit tests.
- Working with small, local mock datasets.
Code Examples
Python (PySpark) Implementation
from pyspark.sql import SparkSession
# Build session in local mode using all CPU cores
spark = SparkSession.builder \
.master("local[*]") \
.appName("LocalModeTest") \
.getOrCreate()
df = spark.range(1, 100)
print("Parallel Partitions:", df.rdd.getNumPartitions())
Expected Output
Parallel Partitions: 8
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.master("local[*]")
.appName("LocalModeScala")
.getOrCreate()
val df = spark.range(1, 100)
println(s"Parallel Partitions: ${df.rdd.getNumPartitions}")
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.