beginner

Aggregations

By DataPlayArena Technical Review BoardUpdated 2026-07-088 min readRuntime Verified

Overview

Learn how to compute statistical summaries—such as sum, average, count, minimum, and maximum—across your Spark DataFrames.

What You Will Learn

In this lesson, you will learn:
  • Summary Functions: Computing sum, avg, min, and max.
  • Agg API: Using the .agg() helper for multiple calculations.
  • DataFrame Stats: Summarizing columns efficiently.

Detailed Concept Explanation

Aggregations compute a single summary value across a collection of rows. Spark provides standard aggregation functions in the pyspark.sql.functions module:

  • sum(): Calculates the total sum.
  • avg() or mean(): Calculates the average.
  • min() and max(): Finds extreme values.
  • count(): Counts non-null records.

You can apply these functions to the entire DataFrame using the .agg() method.


Code Examples

Input Dataset Preview

Below is the sales transaction dataset:

transactionIdamount
1250.0
2150.0
3500.0

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession
from pyspark.sql.functions import sum, avg, max

spark = SparkSession.builder.appName("Aggregations").getOrCreate()
data = [(1, 250.0), (2, 150.0), (3, 500.0)]
df = spark.createDataFrame(data, ["transactionId", "amount"])

# Compute multiple aggregations
summary_df = df.agg(
    sum("amount").alias("total_sales"),
    avg("amount").alias("average_sales"),
    max("amount").alias("max_sales")
)
summary_df.show()

Expected Output

text
+-----------+-------------+---------+
|total_sales|average_sales|max_sales|
+-----------+-------------+---------+
|      900.0|        300.0|    500.0|
+-----------+-------------+---------+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
createDataFrame
agg(sum
avg
max)
show()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

val spark = SparkSession.builder().appName("AggScala").getOrCreate()
import spark.implicits._

val df = Seq((1, 250.0), (2, 150.0), (3, 500.0)).toDF("transactionId", "amount")
val summary = df.agg(
  sum("amount").as("total_sales"),
  avg("amount").as("average_sales"),
  max("amount").as("max_sales")
)
summary.show()

Expected Output

text
+-----------+-------------+---------+
|total_sales|average_sales|max_sales|
+-----------+-------------+---------+
|      900.0|        300.0|    500.0|
+-----------+-------------+---------+

SQL Implementation

sql
SELECT 
    SUM(amount) AS total_sales, 
    AVG(amount) AS average_sales, 
    MAX(amount) AS max_sales 
FROM sales;

Expected Output

Executing this SQL query returns:

total_salesaverage_salesmax_sales
900.0300.0500.0

Common Mistakes

  • Confusing count() Action with count() Function: Calling df.count() (an action that returns an integer row count) is different from count("column") (a transformation function used inside agg() to count non-null values in a column).

Best Practices

  • Alias Aggregations: Always alias your aggregated columns using .alias(). Otherwise, Spark generates default column names like sum(amount).

Interview Perspective

How do you compute multiple aggregations on a DataFrame in Spark?

Use the .agg() method. It allows you to pass a list of aggregation functions (like sum(), avg(), and max()) from pyspark.sql.functions to compute multiple metrics in a single pass.


Interactive Challenges

Challenge 1: Find Minimum Value (Beginner)

Write a PySpark expression to find the minimum value in a column named 'price'.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.