Data Skew
Overview
Learn how to identify data skew (where a single partition holds most of the data) and use salting to distribute records evenly across executors.
What You Will Learn
In this lesson, you will learn:
- Skew Concept: Uneven partition distributions causing straggler tasks.
- Diagnostics: Identifying skew using Web UI stage runtimes.
- Salting Technique: Appending random numbers to keys to split skewed partitions.
Detailed Concept Explanation
Data Skew occurs when data is distributed unevenly across partitions. For example, if a dataset is partitioned by country and 90% of the rows contain 'US', the partition for 'US' will be massive compared to the others.
Why Data Skew hurts
- Straggler Tasks: Spark parallelizes work across partitions. A single massive partition will take hours to process while all other executor cores sit idle.
- OOM Errors: The executor processing the skewed partition can run out of memory (OOM) and crash.
How to resolve Skew: Salting
"Salting" is a technique where you add a random suffix (the "salt") to the join or group keys of your dataset. This splits the skewed keys across multiple partitions.
Skewed Key: 'US' ===> Salted Keys: 'US_0', 'US_1', 'US_2', 'US_3'
To join this salted dataset, you replicate the keys in the lookup table (e.g. duplicating 'US' rows to match all salt suffixes _0 to _3).
Code Examples
Python (PySpark) Implementation (Salting keys)
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, concat, lit, rand
spark = SparkSession.builder.appName("DataSkewSalting").getOrCreate()
# Create skewed DataFrame
df = spark.createDataFrame([("US", 10), ("US", 20), ("FR", 5)], ["country", "val"])
# Add a random salt suffix (0 to 3) to the country key
salt_df = df.withColumn("salt", (rand() * 4).cast("int")) \
.withColumn("salted_country", concat(col("country"), lit("_"), col("salt")))
salt_df.show()
Expected Output
+-------+---+----+--------------+
|country|val|salt|salted_country|
+-------+---+----+--------------+
| US| 10| 2| US_2|
| US| 20| 0| US_0|
| FR| 5| 1| FR_1|
+-------+---+----+--------------+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
val spark = SparkSession.builder().appName("SkewScala").getOrCreate()
import spark.implicits._
val df = Seq(("US", 10), ("US", 20), ("FR", 5)).toDF("country", "val")
val salted = df.withColumn("salt", (rand() * 4).cast("int"))
.withColumn("salted_country", concat($"country", lit("_"), $"salt"))
salted.show()
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.