DataFrame API Reference

Core PySpark DataFrame transformations and selection methods.

select

Return: DataFrame

Projects a set of expressions and returns a new DataFrame.

Syntax signature:df.select(*cols)

Code snippet:

python

df.select('name', 'salary')

Remember: Always project only the columns you need to minimize memory footprint and execution plans complexity.

filter

Return: DataFrame

Filters rows using a given condition or expression.

Syntax signature:df.filter(condition)

Code snippet:

python

df.filter(col('age') > 21)

Remember: Filter data as early as possible in your pipeline to reduce volume before shuffling.

withColumn

Return: DataFrame

Returns a new DataFrame by adding a column or replacing an existing one.

Syntax signature:df.withColumn(colName, colVal)

Code snippet:

python

df.withColumn('tax', col('salary') * 0.2)

Remember: Avoid chaining multiple withColumn calls; use select or selectExpr for bulk column projections.

drop

Return: DataFrame

Returns a new DataFrame with specified columns omitted.

Syntax signature:df.drop(*cols)

Code snippet:

python

df.drop('status', 'bonus')

Remember: Dropping unused columns early prevents carrying useless payloads through shuffle stages.