spark Data Engineering Labs

Learn how to import a CSV file into a Spark DataFrame and filter rows matching a specific threshold condition.

Clean a dirty customers dataset by handling null email columns and deduplicating customer profiles.

Perform basic column additions and math operations to calculate retail prices with tax.

Aggregate employee tables to generate department average and total salary summaries.

Sort employee tables by income to retrieve the top 3 highest-earning workers.

Classify customers into age groups using conditional logic statements.

Join relational employee profiles with department metadata mapping.

Read CSV transaction lists, filter bad records, and write the output as optimized Parquet.

Compute total revenue, units sold, and peak order sizes using general summary statistics.

Apply bonus calculations to employee salaries and rename columns for clear reporting.

Join customer profiles with order histories and run multi-column groupings.

Use Window functions to calculate salary rankings within departments.

Parse timestamp strings to extract year-month groups for chronological reporting.

Identify duplicate transactions occurring within the same minute for the same account.

Compare consecutive purchase values over time using lag functions.

Analyze logical query plans, use caching to prevent redundant execution, and partition data.

Address data skew by salting join keys and using broadcast joins.

Build an incremental ETL pipeline using merge and upsert logic.

Read streaming order feeds and filter fraudulent purchases in real time.

Optimize partitions and reduce shuffle steps in large-scale customer aggregations.