Aggregations Reference

Compute summarized metrics using grouping and aggregations.

groupBy

Return: GroupedData

Groups the DataFrame using the specified columns for aggregation.

Syntax signature:df.groupBy(*cols)

Code snippet:

python

df.groupBy('department').avg('salary')

Remember: groupBy triggers a shuffle stage. Combine with aggressive filters beforehand to limit data transfers.

agg

Return: DataFrame

Computes multiple column aggregations concurrently.

Syntax signature:df.groupBy(...).agg(*exprs)

Code snippet:

python

df.groupBy('dept').agg(max('salary'), sum('bonus'))

Remember: Use agg to calculate distinct, aliases, and custom summaries in a single grouped operation.