Interview Focus: DataFramesDifficulty: Easy

Employee Bonus Calculator

Estimated duration: 10 mins

Lab Overview & Scenario

Apply bonus calculations to employee salaries and rename columns for clear reporting.

Business Scenario

The finance team needs a compensation report. Calculate a standard 10% bonus for each employee earning less than $100,000, and rename the output column to annual_bonus.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

employee_idnamesalary
1Alice120000
2Bob85000
3Charlie95000
4David110000
5Eve70000
View Raw CSV Dataset
employee_id,name,salary
1,Alice,120000
2,Bob,85000
3,Charlie,95000
4,David,110000
5,Eve,70000

2. Your Tasks

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python
# starter.py - Employee Bonus Calculator
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when

def calculate_bonuses():
    spark = SparkSession.builder.appName("BonusCalculator").getOrCreate()
    # TODO: Read dataset.csv, map bonuses with when conditions, rename output
    pass

if __name__ == "__main__":
    calculate_bonuses()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code
Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

dataset.csv
↓
Read CSV
↓
Calculate conditional bonus
↓
Select relevant fields
↓
Display Output

Step-by-Step Implementation Guide

Step 1

Load the employees dataset.

Step 2

Add the column annual_bonus using withColumn().

Step 3

Inside withColumn(), evaluate the condition col("salary") < 100000. If met, calculate col("salary") * 0.1, otherwise return 0.

Step 4

Select columns and print results.

Common Mistakes

  • Returning the bonus as a string instead of maintaining numeric types.
  • Evaluating conditions inside python if/else statements instead of using Spark's when() API.

Interviewer's Expectations

Why this is asked:

Tests basic column calculations combined with conditional updates.

What they look for:

Correct application of Spark conditional when statements instead of python control flow structures.

Common mistakes:

Using python logic loops or list comprehensions, which are not distributed operations.

An excellent answer includes:

Explain that Spark's when() evaluates lazily across partition rows on distributed executor nodes.

✔ What You Learned

  • ✔Map conditional mathematical updates
  • ✔Avoid using python loops for data processing
  • ✔Generate clean compensation reports