Interview Focus: PerformanceDifficulty: Hard

Handle Data Skew

Estimated duration: 20 mins

Lab Overview & Scenario

Address data skew by salting join keys and using broadcast joins.

Business Scenario

A join between a massive clicks dataset and a small lookup table is failing because a few customer IDs hold 90% of the transactions (data skew). Resolve this using broadcast joins.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

user_idaction
1001click
1001view
1001purchase
1002click
1001click
---
user_iduser_name
1001System User
1002Standard User
View Raw CSV Dataset
user_id,action
1001,click
1001,view
1001,purchase
1002,click
1001,click
---
user_id,user_name
1001,System User
1002,Standard User

2. Your Tasks

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python
# starter.py - Handle Data Skew
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, broadcast, concat, lit

def handle_skew():
    spark = SparkSession.builder.appName("DataSkewLab").getOrCreate()
    
    act_data = [(1001, "click"), (1001, "view"), (1001, "purchase"), (1002, "click"), (1001, "click")]
    user_data = [(1001, "System User"), (1002, "Standard User")]
    
    # TODO: Perform broadcast join
    pass

if __name__ == "__main__":
    handle_skew()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code
Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

Raw Mock Lists
↓
Create DataFrames
↓
Apply broadcast() to Users
↓
Add random salt to user_id
↓
Join tables
↓
Display Output

Step-by-Step Implementation Guide

Step 1

Initialize activities and user DataFrames.

Step 2

Wrap the smaller DataFrame with the broadcast() function inside the join statement: act_df.join(broadcast(user_df), on="user_id"). This avoids shuffling the massive activities table by sending a copy of the users table to every executor.

Step 3

Add a salted key to distribute skewed customer ID values across multiple partitions: append a random integer using concat(col("user_id"), lit("_"), (rand() * 2).cast("int")).

Step 4

Print the salted records to verify they are distributed correctly.

Common Mistakes

  • Broadcasting large tables, which can exhaust executor memory and throw OutOfMemory (OOM) errors.
  • Performing standard joins on skewed keys without salting, causing straggler tasks that slow down the entire job.

Interviewer's Expectations

Why this is asked:

Data skew is the most common reason production Spark jobs run slowly or fail with OOM errors.

What they look for:

Understanding of broadcast joins (map-side joins) and the concept of salting join keys.

Common mistakes:

Not explaining *why* skew happens or suggesting resource allocation upgrades instead of optimization.

An excellent answer includes:

Explain that broadcast joins avoid shuffles, and details how salting distributes hot keys across executors.

✔ What You Learned

  • ✔Use broadcast joins to prevent shuffle steps
  • ✔Salt join keys to resolve data skew
  • ✔Distribute hot keys across partitions