beginner
Aggregations
8 min readLast updated: 2026-07-08
Overview
Learn how to compute statistical summaries—such as sum, average, count, minimum, and maximum—across your Spark DataFrames.
What You Will Learn
In this lesson, you will learn:
- Summary Functions: Computing sum, avg, min, and max.
- Agg API: Using the
.agg()helper for multiple calculations. - DataFrame Stats: Summarizing columns efficiently.
Detailed Concept Explanation
Aggregations compute a single summary value across a collection of rows. Spark provides standard aggregation functions in the pyspark.sql.functions module:
sum(): Calculates the total sum.avg()ormean(): Calculates the average.min()andmax(): Finds extreme values.count(): Counts non-null records.
You can apply these functions to the entire DataFrame using the .agg() method.
Code Examples
Input Dataset Preview
Below is the sales transaction dataset:
| transactionId | amount |
|---|---|
| 1 | 250.0 |
| 2 | 150.0 |
| 3 | 500.0 |
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
from pyspark.sql.functions import sum, avg, max
spark = SparkSession.builder.appName("Aggregations").getOrCreate()
data = [(1, 250.0), (2, 150.0), (3, 500.0)]
df = spark.createDataFrame(data, ["transactionId", "amount"])
# Compute multiple aggregations
summary_df = df.agg(
sum("amount").alias("total_sales"),
avg("amount").alias("average_sales"),
max("amount").alias("max_sales")
)
summary_df.show()
Expected Output
text
+-----------+-------------+---------+
|total_sales|average_sales|max_sales|
+-----------+-------------+---------+
| 900.0| 300.0| 500.0|
+-----------+-------------+---------+
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
createDataFrame
agg(sum
avg
max)
show()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
val spark = SparkSession.builder().appName("AggScala").getOrCreate()
import spark.implicits._
val df = Seq((1, 250.0), (2, 150.0), (3, 500.0)).toDF("transactionId", "amount")
val summary = df.agg(
sum("amount").as("total_sales"),
avg("amount").as("average_sales"),
max("amount").as("max_sales")
)
summary.show()
Expected Output
text
+-----------+-------------+---------+
|total_sales|average_sales|max_sales|
+-----------+-------------+---------+
| 900.0| 300.0| 500.0|
+-----------+-------------+---------+
SQL Implementation
sql
SELECT
SUM(amount) AS total_sales,
AVG(amount) AS average_sales,
MAX(amount) AS max_sales
FROM sales;
Expected Output
Executing this SQL query returns:
| total_sales | average_sales | max_sales |
|---|---|---|
| 900.0 | 300.0 | 500.0 |
Common Mistakes
- Confusing count() Action with count() Function: Calling
df.count()(an action that returns an integer row count) is different fromcount("column")(a transformation function used insideagg()to count non-null values in a column).
Best Practices
- Alias Aggregations: Always alias your aggregated columns using
.alias(). Otherwise, Spark generates default column names likesum(amount).
Interview Perspective
How do you compute multiple aggregations on a DataFrame in Spark?
Use the .agg() method. It allows you to pass a list of aggregation functions (like sum(), avg(), and max()) from pyspark.sql.functions to compute multiple metrics in a single pass.