GroupBy Operations

Overview

Learn how to group rows based on values in one or more columns and calculate aggregate statistics for each group.

What You Will Learn

Detailed Concept Explanation

Grouping rows allows you to segment your dataset and analyze categories.

The syntax is: df.groupBy("group_column").agg(sum("agg_column"))

Under the hood, groupBy() is a wide transformation. Spark shuffles rows with matching group keys across the network so that they reside on the same executor node, allowing the final aggregates to be computed.

Code Examples

Input Dataset Preview

Below is the sales categories dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession
from pyspark.sql.functions import sum

spark = SparkSession.builder.appName("GroupBy").getOrCreate()
data = [("Electronics", 1000), ("Office", 150), ("Electronics", 500)]
df = spark.createDataFrame(data, ["category", "revenue"])

# Calculate total revenue per category
grouped_df = df.groupBy("category").agg(sum("revenue").alias("total_revenue"))
grouped_df.show()

Expected Output

+-----------+-------------+
|   category|total_revenue|
+-----------+-------------+
|Electronics|         1500|
|     Office|          150|
+-----------+-------------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

val spark = SparkSession.builder().appName("GroupByScala").getOrCreate()
import spark.implicits._

val df = Seq(("Electronics", 1000), ("Office", 150), ("Electronics", 500)).toDF("category", "revenue")
val grouped = df.groupBy("category").agg(sum("revenue").as("total_revenue"))
grouped.show()

Expected Output

+-----------+-------------+
|   category|total_revenue|
+-----------+-------------+
|Electronics|         1500|
|     Office|          150|
+-----------+-------------+

SQL Implementation

SELECT category, SUM(revenue) AS total_revenue 
FROM sales 
GROUP BY category;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Forgetting the Aggregation Method: Calling df.groupBy("category") directly and attempting to display it. groupBy() returns a GroupedData object, not a DataFrame. You must chain an aggregation method (like .count() or .agg()) to convert it back to a DataFrame.

Best Practices

• Combine Multiple Grouping Keys: You can pass multiple column parameters to groupBy() (e.g. df.groupBy("country", "city")) to group by multiple fields.

Interview Perspective

Interactive Challenges

Related Topics

• Aggregations
• Window Functions
• Shuffle