SPARK Reference Guide

Revision Time: 8 mins

DataFrame API Reference

Comprehensive PySpark DataFrame transformations, column projections, joins, and filtering operations.

select

Return: DataFrame

Projects a set of expressions or column objects and returns a new DataFrame.

Used In: ETL pipelines, Column Pruning, Data Normalization

Syntax signature:df.select(*cols)

Code snippet:

python

df.select('name', col('salary'), (col('salary') * 0.1).alias('bonus'))

Expected Output:

name	salary	bonus
Alice	90000	9000.0
Bob	75000	7500.0

Common Mistakes:Passing column names as a list without unpacking (`df.select(['a', 'b'])` works, but `df.select(col_list)` needs unpacking `*col_list`).

Related Methods: selectExpr, withColumn

Remember: Always project only necessary columns to minimize memory footprint and optimize Catalyst optimizer physical plans.

selectExpr

Return: DataFrame

Projects SQL expressions specified as raw SQL string strings.

Used In: Complex SQL Expressions, Inline Type Casting

Syntax signature:df.selectExpr(*exprs)

Code snippet:

python

df.selectExpr('name', 'salary * 1.1 as new_salary', 'CASE WHEN age >= 18 THEN "Adult" ELSE "Minor" END as category')

Expected Output:

name	new_salary	category
Alice	99000.0	Adult

Common Mistakes:Forgetting SQL syntax rules inside string quotes.

Remember: Ideal for executing inline SQL functions (like CASE WHEN, CAST, CONCAT) without importing `pyspark.sql.functions`.

filter / where

Return: DataFrame

Filters rows using a SQL condition string or PySpark Column boolean expression.

Used In: Data Cleansing, Partition Pruning, Outlier Removal

Syntax signature:df.filter(condition) # Alias: df.where(condition)

Code snippet:

python

df.filter((col('age') >= 21) & (col('department') == 'Engineering'))

Expected Output:

id	age	department
10	28	Engineering

Common Mistakes:Using Python `and`/`or`/`not` instead of bitwise operators `&`/`|`/`~` with parenthesized conditions.

Remember: Filter data as early as possible (Filter Pushdown) to reduce data volume prior to costly shuffle operations.

withColumn

Return: DataFrame

Returns a new DataFrame by adding a new column or replacing an existing column with the same name.

Used In: Feature Engineering, Column Transformation

Syntax signature:df.withColumn(colName, colVal)

Code snippet:

python

df.withColumn('annual_income', col('monthly_income') * 12)

Expected Output:

monthly_income	annual_income
5000	60000

Common Mistakes:Chaining 20+ `withColumn` calls causing Java StackOverflowError during compilation.

Remember: Chaining dozens of consecutive `withColumn` calls creates massive Catalyst logical trees. Prefer a single `select` or `selectExpr` for multi-column additions.

withColumnRenamed

Return: DataFrame

Renames an existing column in the DataFrame.

Used In: Schema Standardization, Data Normalization

Syntax signature:df.withColumnRenamed(existing, new)

Code snippet:

python

df.withColumnRenamed('emp_id', 'employee_id')

Expected Output:

employee_id	name
101	Alice

Remember: Does not fail if the existing column does not exist; it returns the DataFrame unchanged. Use `toDF(*new_cols)` to rename all columns at once.

drop

Return: DataFrame

Returns a new DataFrame with specified column(s) removed.

Used In: Payload Reduction, PII Removal

Syntax signature:df.drop(*cols)

Code snippet:

python

df.drop('temp_col', 'unused_status')

Expected Output:Schema without 'temp_col' and 'unused_status'

Remember: Dropping unused columns early prevents carrying useless payloads through shuffle stages and parquet output files.

dropDuplicates / distinct

Return: DataFrame

Removes duplicate rows from the DataFrame, optionally considering only a subset of columns.

Used In: Deduplication, Primary Key Enforcement

Syntax signature:df.dropDuplicates(subset=None) # distinct() checks all columns

Code snippet:

python

df.dropDuplicates(subset=['email'])

Expected Output:Unique rows per email address

Common Mistakes:Assuming `dropDuplicates()` preserves ordering. Use Window functions (`row_number()`) if ordering matters.

Remember: Triggers a full network shuffle across partitions. Ensure dataset is filtered before deduplication.

join

Return: DataFrame

Joins with another DataFrame using the specified join key and strategy.

Used In: Table Merging, Relational Data Lookups

Syntax signature:df1.join(df2, on=..., how='inner')

Code snippet:

python

df1.join(df2, df1.user_id == df2.user_id, how='left')

Expected Output:Merged DataFrame containing combined columns

Common Mistakes:Ambiguous column names after join (`df1.user_id` and `df2.user_id` both present). Pass `on='user_id'` as a string to drop duplicate join key automatically.

Remember: Supported join types: 'inner', 'left', 'right', 'full', 'semi', 'anti', 'cross'. Use broadcast join (`broadcast(small_df)`) when one side is under 10MB.

Next TopicRDD API Reference