Local Mode

Overview

Learn how to run Spark in Local Mode on a single machine for fast development, code testing, and prototyping.

What You Will Learn

In this lesson, you will learn:

Local Mode Architecture: Driver and executor executing in a single local JVM.
Master String: Configuring threads using local[N] master configurations.
Development Cycles: Fast local testing cycles.

Detailed Concept Explanation

Local Mode is the simplest deployment mode in Apache Spark. Instead of launching processes across a distributed network of nodes, Spark runs the Driver, executors, and master threads inside a single JVM on your local machine.

Master Configuration Values

You configure the local thread count by setting the master parameter:

local: Runs Spark on a single thread. (Slowest, no parallel execution).
local[4]: Runs Spark using exactly 4 execution threads.
local[*]: Runs Spark using as many threads as there are logical CPU cores on your machine.

When to use it

Code testing and local prototyping.
Running unit tests.
Working with small, local mock datasets.

Code Examples

Python (PySpark) Implementation

python

from pyspark.sql import SparkSession

# Build session in local mode using all CPU cores
spark = SparkSession.builder \
    .master("local[*]") \
    .appName("LocalModeTest") \
    .getOrCreate()

df = spark.range(1, 100)
print("Parallel Partitions:", df.rdd.getNumPartitions())

Expected Output

text

Parallel Partitions: 8

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

master(local[*])

appName(LocalModeTest)

range(1

100)

getNumPartitions()

Scala Implementation

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .master("local[*]")
  .appName("LocalModeScala")
  .getOrCreate()

val df = spark.range(1, 100)
println(s"Parallel Partitions: ${df.rdd.getNumPartitions}")