intermediate

Spark Configuration

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how to configure Spark properties using SparkConf, configuration files, and environment variables.

What You Will Learn

In this lesson, you will learn:
  • Configuration Precedence: How config values override each other.
  • SparkConf API: Setting configurations programmatically inside code.
  • Configuration Files: Setting defaults using spark-defaults.conf.

Detailed Concept Explanation

Spark provides a highly customizable configuration system. Spark properties control everything from executor memory allocations to query join strategies.

Order of Configuration Precedence

If a property is configured in multiple places, Spark resolves them in the following order of priority:

  1. Dynamic Code Setups (Highest): Properties set directly inside the application code using SparkConf (e.g. .config("key", "value")).
  2. Command Line Flags: Values passed to spark-submit at runtime (e.g. --executor-memory 4G).
  3. Properties Configuration File: Defaults set in conf/spark-defaults.conf on the driver node.
  4. Environment Variables (Lowest): System variables set in conf/spark-env.sh (e.g. SPARK_LOCAL_IP).

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

# Define properties during session building
spark = SparkSession.builder \
    .appName("ConfigTest") \
    .config("spark.sql.shuffle.partitions", "50") \
    .config("spark.executor.memory", "4g") \
    .getOrCreate()

# Retrieve property at runtime
print("Shuffle Partitions:", spark.conf.get("spark.sql.shuffle.partitions"))

Expected Output

text
Shuffle Partitions: 50

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
config(shuffle.partitions
50)
config(executor.memory
4g)
getOrCreate()
conf.get(shuffle.partitions)

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("ConfigScala")
  .config("spark.sql.shuffle.partitions", "50")
  .config("spark.executor.memory", "4g")
  .getOrCreate()

println(s"Shuffle Partitions: ${spark.conf.get("spark.sql.shuffle.partitions")}")

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.