Convert CSV to Parquet

Business Scenario

To optimize warehouse query costs, you must write a pipeline that reads transactional CSV logs, filters out negative amount lines, and saves the cleaned logs as columnar Parquet files.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

txn_id,account_id,amount
3001,1001,150.0
3002,1002,-50.0
3003,1003,300.0
3004,1004,0.0
3005,1005,450.0

2. Your Tasks

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

# starter.py - Convert CSV to Parquet
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

def process_formats():
    spark = SparkSession.builder.appName("FormatConversion").getOrCreate()
    # TODO: Read dataset.csv, filter amounts > 0, write parquet
    pass

if __name__ == "__main__":
    process_formats()

Step-by-Step Implementation Guide