Interview Focus: PerformanceDifficulty: Hard

Handle Data Skew

Estimated duration: 20 mins

Lab Overview & Scenario

Address data skew by salting join keys and using broadcast joins.

Business Scenario

A join between a massive clicks dataset and a small lookup table is failing because a few customer IDs hold 90% of the transactions (data skew). Resolve this using broadcast joins.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

user_id	action
1001	click
1001	view
1001	purchase
1002	click
1001	click
---
user_id	user_name
1001	System User
1002	Standard User

View Raw CSV Dataset

user_id,action
1001,click
1001,view
1001,purchase
1002,click
1001,click
---
user_id,user_name
1001,System User
1002,Standard User

2. Your Tasks

Simulate two DataFrames (Activities and Users).

Use a broadcast join to copy the small Users lookup table to all executors, avoiding a full shuffle.

Demonstrate how salting works: append a random integer to join keys to distribute partition records.

Perform the join and show the results.

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python

# starter.py - Handle Data Skew
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, broadcast, concat, lit

def handle_skew():
    spark = SparkSession.builder.appName("DataSkewLab").getOrCreate()
    
    act_data = [(1001, "click"), (1001, "view"), (1001, "purchase"), (1002, "click"), (1001, "click")]
    user_data = [(1001, "System User"), (1002, "Standard User")]
    
    # TODO: Perform broadcast join
    pass

if __name__ == "__main__":
    handle_skew()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code

Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

Raw Mock Lists

↓

Create DataFrames

↓

Apply broadcast() to Users

↓

Add random salt to user_id

↓

Join tables

↓

Display Output

Step-by-Step Implementation Guide

Step 1

Initialize activities and user DataFrames.

Step 2

Wrap the smaller DataFrame with the broadcast() function inside the join statement: act_df.join(broadcast(user_df), on="user_id"). This avoids shuffling the massive activities table by sending a copy of the users table to every executor.

Step 3

Add a salted key to distribute skewed customer ID values across multiple partitions: append a random integer using concat(col("user_id"), lit("_"), (rand() * 2).cast("int")).

Step 4

Print the salted records to verify they are distributed correctly.

Common Mistakes

Broadcasting large tables, which can exhaust executor memory and throw OutOfMemory (OOM) errors.
Performing standard joins on skewed keys without salting, causing straggler tasks that slow down the entire job.

Interviewer's Expectations

Why this is asked:

Data skew is the most common reason production Spark jobs run slowly or fail with OOM errors.

What they look for:

Understanding of broadcast joins (map-side joins) and the concept of salting join keys.

Common mistakes:

Not explaining *why* skew happens or suggesting resource allocation upgrades instead of optimization.

An excellent answer includes:

Explain that broadcast joins avoid shuffles, and details how salting distributes hot keys across executors.

✔ What You Learned

✔Use broadcast joins to prevent shuffle steps
✔Salt join keys to resolve data skew
✔Distribute hot keys across partitions

Continue Learning

Previous LabOptimize a Slow Spark Job Related Theory LessonData Skew Next LabIncremental ETL Pipeline