DataPlayArena
Interview Focus: DataFramesDifficulty: Easy

Customer Age Categories

Estimated duration: 10 mins

Lab Overview & Scenario

Classify customers into age groups using conditional logic statements.

Business Scenario

The marketing department wants to categorize customers into age demographics for tailored marketing campaigns. Classify users under 30 as 'Young', 30-50 as 'Adult', and others as 'Senior'.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

customer_idnameage
1Alice25
2Bob45
3Charlie60
4David30
5Eve22
View Raw CSV Dataset
customer_id,name,age
1,Alice,25
2,Bob,45
3,Charlie,60
4,David,30
5,Eve,22

2. Your Tasks

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python
# starter.py - Customer Age Categories
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when

def categorize_customers():
    spark = SparkSession.builder.appName("AgeCategories").getOrCreate()
    # TODO: Read dataset.csv, apply conditional when statements
    pass

if __name__ == "__main__":
    categorize_customers()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code
Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

dataset.csv
↓
Read CSV
↓
Apply when/otherwise conditions
↓
Select columns
↓
Display Output

Step-by-Step Implementation Guide

Step 1

Load the customers dataset with age parsed as an integer.

Step 2

Add a new column called age_group.

Step 3

Use when(condition, value) to map matching values. Chain further conditional matches with additional .when() calls, and finalize using .otherwise() to handle default cases.

Step 4

Select and print name, age, and age group columns.

Common Mistakes

  • Forgetting to wrap individual conditions in parentheses inside a compound condition: e.g. col('age') >= 30 & col('age') <= 50 instead of (col('age') >= 30) & (col('age') <= 50).
  • Using python raw word statements 'and' / 'or' which will raise syntax evaluation errors.

Interviewer's Expectations

Why this is asked:

Conditional transformations (equivalent to SQL CASE WHEN) are crucial for feature engineering.

What they look for:

Clean execution of nested when() statements and proper boolean condition compounding.

Common mistakes:

Using python keywords (and, or) instead of Spark's bitwise operators (&, |).

An excellent answer includes:

Write the logic using chaining, explaining how Spark evaluates when blocks sequentially.

✔ What You Learned

  • ✔Execute conditional code blocks using when().otherwise()
  • ✔Write compound boolean expressions with logical operators
  • ✔Map values based on business categorization rules

Continue Learning

Previous LabTop Paid EmployeesRelated Theory LessonBuilt-in FunctionsNext LabBasic DataFrame Joins