intermediate
Spark Configuration
Overview
Learn how to configure Spark properties using SparkConf, configuration files, and environment variables.
What You Will Learn
In this lesson, you will learn:
- Configuration Precedence: How config values override each other.
- SparkConf API: Setting configurations programmatically inside code.
- Configuration Files: Setting defaults using
spark-defaults.conf.
Detailed Concept Explanation
Spark provides a highly customizable configuration system. Spark properties control everything from executor memory allocations to query join strategies.
Order of Configuration Precedence
If a property is configured in multiple places, Spark resolves them in the following order of priority:
- Dynamic Code Setups (Highest): Properties set directly inside the application code using
SparkConf(e.g..config("key", "value")). - Command Line Flags: Values passed to
spark-submitat runtime (e.g.--executor-memory 4G). - Properties Configuration File: Defaults set in
conf/spark-defaults.confon the driver node. - Environment Variables (Lowest): System variables set in
conf/spark-env.sh(e.g.SPARK_LOCAL_IP).
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
# Define properties during session building
spark = SparkSession.builder \
.appName("ConfigTest") \
.config("spark.sql.shuffle.partitions", "50") \
.config("spark.executor.memory", "4g") \
.getOrCreate()
# Retrieve property at runtime
print("Shuffle Partitions:", spark.conf.get("spark.sql.shuffle.partitions"))
Expected Output
text
Shuffle Partitions: 50
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
config(shuffle.partitions
50)
config(executor.memory
4g)
getOrCreate()
conf.get(shuffle.partitions)
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder()
.appName("ConfigScala")
.config("spark.sql.shuffle.partitions", "50")
.config("spark.executor.memory", "4g")
.getOrCreate()
println(s"Shuffle Partitions: ${spark.conf.get("spark.sql.shuffle.partitions")}")
Related Topics
DPA
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.