Performance Tuning Checklist

Overview

A comprehensive checklist and troubleshooting guide to diagnose, debug, and optimize slow Spark pipelines.

What You Will Learn

In this lesson, you will learn:

Diagnostic Guide: Checking Spark Web UI runtime metrics.
Memory Configurations: Tuning executor allocation flags.
Checklist: Optimization best practices to apply to your pipelines.

Detailed Concept Explanation

Use this checklist when tuning and optimizing your Spark pipelines:

1. File Formats & Reader Optimization

[ ] Are you storing analytical datasets in columnar formats like Parquet, ORC, or Delta Lake instead of CSV or JSON?
[ ] Have you enabled Predicate Pushdown and Partition Pruning by applying filters directly after reading?
[ ] If loading from JDBC databases, are you partitioning reads using bounds to parallelize connection fetches?

2. Join & Aggregation Strategies

[ ] Are you using Broadcast Hash Joins (broadcast(small_df)) when joining a large dataset with lookup tables under 10MB?
[ ] Is Adaptive Query Execution (AQE) enabled (spark.sql.adaptive.enabled = true) to handle small partition merges dynamically?
[ ] When using the RDD API, are you using reduceByKey or aggregateByKey instead of groupByKey to enable map-side combining?

3. Partitioning & Layout Controls

[ ] Are your partition sizes balanced, targeting an uncompressed size of 100MB-200MB in memory?
[ ] Have you coalesced partition layouts (df.coalesce(n)) before writing outputs to disk to prevent the Small File Problem?
[ ] Have you applied salting techniques to redistribute skewed join keys evenly across executors?

4. Memory & Serialization Settings

[ ] Is Kryo serialization enabled (spark.serializer = org.apache.spark.serializer.KryoSerializer)?
[ ] Have you unpersisted cached DataFrames (df.unpersist()) once they are no longer needed to free up executor RAM?

Code Examples

Python (PySpark) Implementation (Applying core optimization configurations)

python

from pyspark.sql import SparkSession

# Build optimal production session settings
spark = SparkSession.builder \
    .appName("ProductionTuning") \
    .config("spark.sql.adaptive.enabled", "true") \
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
    .config("spark.sql.autoBroadcastJoinThreshold", "10485760") \
    .getOrCreate()

print("Production Tuning Configurations Applied.")

Expected Output

text

Production Tuning Configurations Applied.

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

config(adaptive

true)

config(serializer

KryoSerializer)

config(autoBroadcastJoinThreshold

10MB)

getOrCreate()

Scala Implementation

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("ProductionTuningScala")
  .config("spark.sql.adaptive.enabled", "true")
  .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .config("spark.sql.autoBroadcastJoinThreshold", "10485760")
  .getOrCreate()

println("Production Tuning Configurations Applied.")