Interview Focus: PerformanceDifficulty: Hard

Large-Scale Customer Analytics

Estimated duration: 20 mins

Lab Overview & Scenario

Optimize partitions and reduce shuffle steps in large-scale customer aggregations.

Business Scenario

A large customer aggregate query is shuffling terabytes of data across the network, causing bottleneck delays. Optimize this by tuning the partition size and reducing shuffles.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

customer_id	spent
101	15.0
102	22.0
101	8.0
103	11.0
102	15.0

View Raw CSV Dataset

customer_id,spent
101,15.0
102,22.0
101,8.0
103,11.0
102,15.0

2. Your Tasks

Read the raw sales data.

Set the default shuffle partitions configuration to a size matching the dataset size.

Run aggregations to sum transaction spent values.

Verify partition count changes.

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python

# starter.py - Large-Scale Customer Analytics
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum

def optimize_shuffle():
    spark = SparkSession.builder.appName("ShuffleOptimization").getOrCreate()
    # TODO: Set spark.sql.shuffle.partitions, group & sum, view partition counts
    pass

if __name__ == "__main__":
    optimize_shuffle()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code

Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

dataset.csv

↓

Read CSV

↓

Group by customer_id

↓

Aggregate sum spent

↓

Post-shuffle partition check

↓

Display Output

Step-by-Step Implementation Guide

Step 1

Configure Spark when building the session: set the shuffle partition count to a value that matches the dataset size: .config("spark.sql.shuffle.partitions", "4").

Step 2

Load the transactions dataset.

Step 3

Group by customer_id and calculate the sum of transaction spent amounts.

Step 4

Print the aggregated totals and display the post-shuffle partition count using grouped.rdd.getNumPartitions().

Common Mistakes

Leaving the default value of 200 shuffle partitions for small workloads, causing Spark to waste time creating and managing empty tasks.
Setting partition counts too low on large datasets, leading to out-of-memory errors on executors.

Interviewer's Expectations

Why this is asked:

Tuning shuffles and partition counts is critical for building cost-effective production pipelines.

What they look for:

Understanding of partition counts, executor counts, and shuffle configurations.

Common mistakes:

Not knowing how to change partition counts, or setting static partition counts that don't scale.

An excellent answer includes:

Explain that shuffles default to 200 partitions, why that causes overhead on small datasets, and how Adaptive Query Execution (AQE) dynamically sizes partitions.

✔ What You Learned

✔Configure Spark shuffle parameters
✔Analyze post-shuffle partition counts
✔Optimize partition sizes for small vs large datasets

Continue Learning

Previous LabStreaming Order Processing Related Theory LessonShuffle Manager

Next LabEnd of Track