DataPlayArena
Interview Focus: DataFramesDifficulty: Easy

Top Paid Employees

Estimated duration: 10 mins

Lab Overview & Scenario

Sort employee tables by income to retrieve the top 3 highest-earning workers.

Business Scenario

HR wants to audit high-earner compensation. You are tasked with retrieving profiles of the top 3 highest-paid employees in descending order of their salaries.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

idnamesalary
1Alice120000
2Bob90000
3Charlie150000
4David85000
5Eve130000
View Raw CSV Dataset
id,name,salary
1,Alice,120000
2,Bob,90000
3,Charlie,150000
4,David,85000
5,Eve,130000

2. Your Tasks

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python
# starter.py - Top Paid Employees
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

def top_paid():
    spark = SparkSession.builder.appName("TopPaid").getOrCreate()
    # TODO: Read dataset.csv, sort by salary desc, limit 3
    pass

if __name__ == "__main__":
    top_paid()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code
Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

dataset.csv
↓
Read CSV
↓
Sort salary DESC
↓
Select name, salary
↓
Limit 3
↓
Display Output

Step-by-Step Implementation Guide

Step 1

Load the CSV employees dataset with schema inference.

Step 2

Apply orderBy(col("salary").desc()) to sort the rows from highest salary to lowest.

Step 3

Select name and salary to keep the payload lightweight.

Step 4

Apply limit(3) to extract only the top 3 records, and print the results using show().

Common Mistakes

  • Sorting in ascending order (default), which returns the lowest paid employees instead.
  • Using python slice syntax [0:3] instead of calling Spark's limit(3) method.

Interviewer's Expectations

Why this is asked:

Retrieving extreme limits (max/min top lists) is a typical pattern in SQL/Spark.

What they look for:

Correct application of sorting order and limit operators.

Common mistakes:

Using memory-intensive sorting steps without restricting final records.

An excellent answer includes:

Explain that sorting requires shuffling and that limiting early helps minimize memory usage.

✔ What You Learned

  • ✔Sort DataFrames using orderBy()
  • ✔Configure sort directions using col().desc()
  • ✔Restrict output row counts using limit()

Continue Learning

Previous LabDepartment Salary ReportRelated Theory LessonSpark SQL BasicsNext LabCustomer Age Categories