Sorting Data

Overview

Learn how to sort and order DataFrame rows by one or more columns in ascending or descending order.

What You Will Learn

Detailed Concept Explanation

Sorting reorganizes DataFrame rows based on the values in one or more columns. You can use either sort() or orderBy()—both are identical aliases in Spark.

By default, Spark sorts data in ascending order. To sort in descending order, you must wrap the column reference in desc().

Code Examples

Input Dataset Preview

Below is the student test scores dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("Sorting").getOrCreate()
data = [("Alice", 85), ("Bob", 95), ("Charlie", 90)]
df = spark.createDataFrame(data, ["name", "score"])

# Order by score descending
sorted_df = df.orderBy(col("score").desc())
sorted_df.show()

Expected Output

+-------+-----+
|   name|score|
+-------+-----+
|    Bob|   95|
|Charlie|   90|
|  Alice|   85|
+-------+-----+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("SortScala").getOrCreate()
import spark.implicits._

val df = Seq(("Alice", 85), ("Bob", 95), ("Charlie", 90)).toDF("name", "score")
val sorted = df.orderBy($"score".desc)
sorted.show()

Expected Output

+-------+-----+
|   name|score|
+-------+-----+
|    Bob|   95|
|Charlie|   90|
|  Alice|   85|
+-------+-----+

SQL Implementation

SELECT name, score 
FROM students 
ORDER BY score DESC;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Assuming Sorting is Low-Cost: Sorting requires data to be compared across all partitions, which triggers a wide dependency shuffle. Avoid sorting intermediate datasets in your pipeline; only sort at the very end when displaying or writing final results.

Best Practices

• Sort at the End: Only apply sorting at the end of your pipeline before writing or outputting data, saving unnecessary shuffles.

Interview Perspective

Interactive Challenges

Related Topics

• Filtering Data
• Selecting Columns
• Window Functions