Interview Focus: DataFramesDifficulty: Easy

Employee Data Filtering

Estimated duration: 10 mins

Lab Overview & Scenario

Learn how to import a CSV file into a Spark DataFrame and filter rows matching a specific threshold condition.

Business Scenario

The HR department needs a quick report identifying all high-earning employees in the organization who earn more than $80,000 annually. You must extract only their names and departments.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

id	name	department	salary	active
1	Alice	Engineering	120000	true
2	Bob	Marketing	90000	true
3	Charlie	Sales	75000	true
4	David	Engineering	85000	true
5	Eve	Marketing	78000	true

View Raw CSV Dataset

id,name,department,salary,active
1,Alice,Engineering,120000,true
2,Bob,Marketing,90000,true
3,Charlie,Sales,75000,true
4,David,Engineering,85000,true
5,Eve,Marketing,78000,true

2. Your Tasks

Read the raw employees CSV dataset.

Filter employees with salary greater than 80000.

Select only the name and department columns.

Display the final result using show().

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python

# starter.py - Employee Data Filtering
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

def filter_employees():
    spark = SparkSession.builder.appName("EmployeeFilter").getOrCreate()
    # TODO: Read dataset.csv
    # TODO: Filter salaries > 80000 and select name, department
    pass

if __name__ == "__main__":
    filter_employees()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code

Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

dataset.csv

↓

Read CSV

↓

Filter salaries > 80000

↓

Select name, department

↓

Display Output

Step-by-Step Implementation Guide

Step 1

Read the employees CSV dataset using spark.read.csv specifying header=True and inferSchema=True. This tells Spark to parse the first line as column names and detect numeric fields automatically.

Step 2

Apply the filter operation using col("salary") > 80000 to filter out rows containing lower-paid employees.

Step 3

Select only the name and department columns to match HR's reporting layout.

Step 4

Invoke show() to print the final DataFrame rows to stdout.

Common Mistakes

Not setting inferSchema=True, which makes Spark read salary as a string, causing numeric comparisons to fail.
Using SQL strings instead of PySpark col() expressions in filters without registerTempView.

Interviewer's Expectations

Why this is asked:

To test your absolute core PySpark setup, reading CSV inputs, and basic DataFrame actions.

What they look for:

Understanding of schema inference and proper usage of DataFrame transformation chaining.

Common mistakes:

Reading columns as strings due to missing inferSchema=True.

An excellent answer includes:

Write readable, clean method chains and explain how schema inference works in Spark.

✔ What You Learned

✔Load CSV records with schema detection
✔Perform row filter conditions
✔Select specific columns
✔Display DataFrame outputs in terminal console

Continue Learning

Previous LabStart of Track