Spark Configuration

Overview

Learn how to configure Spark properties using SparkConf, configuration files, and environment variables.

What You Will Learn

Detailed Concept Explanation

Spark provides a highly customizable configuration system. Spark properties control everything from executor memory allocations to query join strategies.

Order of Configuration Precedence

If a property is configured in multiple places, Spark resolves them in the following order of priority:

1. Dynamic Code Setups (Highest): Properties set directly inside the application code using SparkConf (e.g. .config("key", "value")).
2. Command Line Flags: Values passed to spark-submit at runtime (e.g. --executor-memory 4G).
3. Properties Configuration File: Defaults set in conf/spark-defaults.conf on the driver node.
4. Environment Variables (Lowest): System variables set in conf/spark-env.sh (e.g. SPARK_LOCAL_IP).

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

# Define properties during session building
spark = SparkSession.builder \
    .appName("ConfigTest") \
    .config("spark.sql.shuffle.partitions", "50") \
    .config("spark.executor.memory", "4g") \
    .getOrCreate()

# Retrieve property at runtime
print("Shuffle Partitions:", spark.conf.get("spark.sql.shuffle.partitions"))

Expected Output

Shuffle Partitions: 50

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("ConfigScala")
  .config("spark.sql.shuffle.partitions", "50")
  .config("spark.executor.memory", "4g")
  .getOrCreate()

println(s"Shuffle Partitions: ${spark.conf.get("spark.sql.shuffle.partitions")}")

Related Topics

• spark-submit
• Dynamic Resource Allocation
• Memory Management