Data Skew

Overview

Learn how to identify data skew (where a single partition holds most of the data) and use salting to distribute records evenly across executors.

What You Will Learn

In this lesson, you will learn:

Skew Concept: Uneven partition distributions causing straggler tasks.
Diagnostics: Identifying skew using Web UI stage runtimes.
Salting Technique: Appending random numbers to keys to split skewed partitions.

Detailed Concept Explanation

Data Skew occurs when data is distributed unevenly across partitions. For example, if a dataset is partitioned by country and 90% of the rows contain 'US', the partition for 'US' will be massive compared to the others.

Why Data Skew hurts

Straggler Tasks: Spark parallelizes work across partitions. A single massive partition will take hours to process while all other executor cores sit idle.
OOM Errors: The executor processing the skewed partition can run out of memory (OOM) and crash.

How to resolve Skew: Salting

"Salting" is a technique where you add a random suffix (the "salt") to the join or group keys of your dataset. This splits the skewed keys across multiple partitions.

text

Skewed Key: 'US'  ===>  Salted Keys: 'US_0', 'US_1', 'US_2', 'US_3'

To join this salted dataset, you replicate the keys in the lookup table (e.g. duplicating 'US' rows to match all salt suffixes _0 to _3).

Code Examples

Python (PySpark) Implementation (Salting keys)

python

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, concat, lit, rand

spark = SparkSession.builder.appName("DataSkewSalting").getOrCreate()

# Create skewed DataFrame
df = spark.createDataFrame([("US", 10), ("US", 20), ("FR", 5)], ["country", "val"])

# Add a random salt suffix (0 to 3) to the country key
salt_df = df.withColumn("salt", (rand() * 4).cast("int")) \
            .withColumn("salted_country", concat(col("country"), lit("_"), col("salt")))

salt_df.show()

Expected Output

text

+-------+---+----+--------------+
|country|val|salt|salted_country|
+-------+---+----+--------------+
|     US| 10|   2|          US_2|
|     US| 20|   0|          US_0|
|     FR|  5|   1|          FR_1|
+-------+---+----+--------------+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

createDataFrame

withColumn(salt

rand)

withColumn(salted_country)

show()

Scala Implementation

scala

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

val spark = SparkSession.builder().appName("SkewScala").getOrCreate()
import spark.implicits._

val df = Seq(("US", 10), ("US", 20), ("FR", 5)).toDF("country", "val")

val salted = df.withColumn("salt", (rand() * 4).cast("int"))
  .withColumn("salted_country", concat($"country", lit("_"), $"salt"))

salted.show()