DataPlayArena
Interview Focus: DataFramesDifficulty: Easy

Product Price Analysis

Estimated duration: 10 mins

Lab Overview & Scenario

Perform basic column additions and math operations to calculate retail prices with tax.

Business Scenario

A retail company wants to list prices containing an added 15% VAT tax for all electronic products. You need to calculate the final price and filter to list only Electronics.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

product_idproduct_namecategoryprice
201LaptopElectronics1000.0
202Desk ChairFurniture150.0
203HeadphonesElectronics120.0
204Coffee MakerAppliances80.0
205Smart WatchElectronics250.0
View Raw CSV Dataset
product_id,product_name,category,price
201,Laptop,Electronics,1000.0
202,Desk Chair,Furniture,150.0
203,Headphones,Electronics,120.0
204,Coffee Maker,Appliances,80.0
205,Smart Watch,Electronics,250.0

2. Your Tasks

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python
# starter.py - Product Price Analysis
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

def analyze_products():
    spark = SparkSession.builder.appName("ProductAnalysis").getOrCreate()
    # TODO: Read dataset.csv, add VAT tax column, filter Electronics
    pass

if __name__ == "__main__":
    analyze_products()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code
Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

dataset.csv
↓
Read CSV
↓
Add VAT (price * 1.15)
↓
Filter Electronics
↓
Display Output

Step-by-Step Implementation Guide

Step 1

Load the CSV products dataset with types inferred so price is double precision.

Step 2

Add the column price_with_vat by applying df.withColumn() and multiplying the price column by 1.15.

Step 3

Filter the rows using equality test col("category") == "Electronics".

Step 4

Select and display name, category, and final price.

Common Mistakes

  • Forgetting to load columns as doubles, resulting in string multiplications returning errors.
  • Trying to reassign columns using standard brackets df['new_col'] = ... which is not supported in Spark.

Interviewer's Expectations

Why this is asked:

Column operations and basic mathematical maps are essential to build ETL pipelines.

What they look for:

Proper usage of withColumn and columns arithmetic syntax.

Common mistakes:

Trying to mutate DataFrames in place (Spark DataFrames are immutable).

An excellent answer includes:

Explain that withColumn creates a new DataFrame projection without changing physical layout.

✔ What You Learned

  • ✔Add columns using withColumn()
  • ✔Perform arithmetic operations on numeric columns
  • ✔Chain filters and column selections

Continue Learning

Previous LabCustomer Data CleaningRelated Theory LessonSelecting ColumnsNext LabDepartment Salary Report