Interview Focus: DataFramesDifficulty: Easy

Employee Bonus Calculator

Estimated duration: 10 mins

Lab Overview & Scenario

Apply bonus calculations to employee salaries and rename columns for clear reporting.

Business Scenario

The finance team needs a compensation report. Calculate a standard 10% bonus for each employee earning less than $100,000, and rename the output column to annual_bonus.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

employee_id	name	salary
1	Alice	120000
2	Bob	85000
3	Charlie	95000
4	David	110000
5	Eve	70000

View Raw CSV Dataset

employee_id,name,salary
1,Alice,120000
2,Bob,85000
3,Charlie,95000
4,David,110000
5,Eve,70000

2. Your Tasks

Read the raw employees CSV dataset.

Add a column calculated as: if salary < 100000 then salary * 0.1 else 0.

Rename the bonus column to 'annual_bonus'.

Select and print only name, salary, and annual_bonus.

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python

# starter.py - Employee Bonus Calculator
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, when

def calculate_bonuses():
    spark = SparkSession.builder.appName("BonusCalculator").getOrCreate()
    # TODO: Read dataset.csv, map bonuses with when conditions, rename output
    pass

if __name__ == "__main__":
    calculate_bonuses()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code

Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

dataset.csv

↓

Read CSV

↓

Calculate conditional bonus

↓

Select relevant fields

↓

Display Output

Step-by-Step Implementation Guide

Step 1

Load the employees dataset.

Step 2

Add the column annual_bonus using withColumn().

Step 3

Inside withColumn(), evaluate the condition col("salary") < 100000. If met, calculate col("salary") * 0.1, otherwise return 0.

Step 4

Select columns and print results.

Common Mistakes

Returning the bonus as a string instead of maintaining numeric types.
Evaluating conditions inside python if/else statements instead of using Spark's when() API.

Interviewer's Expectations

Why this is asked:

Tests basic column calculations combined with conditional updates.

What they look for:

Correct application of Spark conditional when statements instead of python control flow structures.

Common mistakes:

Using python logic loops or list comprehensions, which are not distributed operations.

An excellent answer includes:

Explain that Spark's when() evaluates lazily across partition rows on distributed executor nodes.

✔ What You Learned

✔Map conditional mathematical updates
✔Avoid using python loops for data processing
✔Generate clean compensation reports

Continue Learning

Previous LabSales Summary Report Related Theory LessonBuilt-in Functions Next LabCustomer Order Analytics