beginner

Local Mode

By DataPlayArena Technical Review BoardUpdated 2026-07-096 min readRuntime Verified

Overview

Learn how to run Spark in Local Mode on a single machine for fast development, code testing, and prototyping.

What You Will Learn

In this lesson, you will learn:
  • Local Mode Architecture: Driver and executor executing in a single local JVM.
  • Master String: Configuring threads using local[N] master configurations.
  • Development Cycles: Fast local testing cycles.

Detailed Concept Explanation

Local Mode is the simplest deployment mode in Apache Spark. Instead of launching processes across a distributed network of nodes, Spark runs the Driver, executors, and master threads inside a single JVM on your local machine.

Master Configuration Values

You configure the local thread count by setting the master parameter:

  • local: Runs Spark on a single thread. (Slowest, no parallel execution).
  • local[4]: Runs Spark using exactly 4 execution threads.
  • local[*]: Runs Spark using as many threads as there are logical CPU cores on your machine.

When to use it

  • Code testing and local prototyping.
  • Running unit tests.
  • Working with small, local mock datasets.

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

# Build session in local mode using all CPU cores
spark = SparkSession.builder \
    .master("local[*]") \
    .appName("LocalModeTest") \
    .getOrCreate()

df = spark.range(1, 100)
print("Parallel Partitions:", df.rdd.getNumPartitions())

Expected Output

text
Parallel Partitions: 8

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
master(local[*])
appName(LocalModeTest)
range(1
100)
getNumPartitions()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .master("local[*]")
  .appName("LocalModeScala")
  .getOrCreate()

val df = spark.range(1, 100)
println(s"Parallel Partitions: ${df.rdd.getNumPartitions}")

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.