Marketing wants to send promotional emails but the database has duplicates and missing values. You must fill null emails with 'N/A' and deduplicate records based on email addresses.
The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):
| customer_id | name | |
|---|---|---|
| 101 | John Doe | john@example.com |
| 102 | Jane Smith | |
| 103 | John Doe | john@example.com |
| 104 | Alice Jones | alice@example.com |
| 105 | Jane Smith | jane@example.com |
customer_id,name,email 101,John Doe,john@example.com 102,Jane Smith, 103,John Doe,john@example.com 104,Alice Jones,alice@example.com 105,Jane Smith,jane@example.com
Copy this starter template to write the data transformation logic:
# starter.py - Customer Data Cleaning
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
def clean_customers():
spark = SparkSession.builder.appName("CustomerCleaning").getOrCreate()
# TODO: Read dataset.csv, fill null emails, drop duplicate emails
pass
if __name__ == "__main__":
clean_customers()df.na.fill() to replace empty or null string values inside the email column with "N/A".
dropDuplicates(["email"]) to remove repeated records, ensuring each email address appears exactly once.