Sorting Data
Overview
Learn how to sort and order DataFrame rows by one or more columns in ascending or descending order.
What You Will Learn
In this lesson, you will learn:
- Sort API: Using
sort()andorderBy(). - Ascending vs Descending: Toggling sort directions (
asc()anddesc()). - Multi-column Sorting: Ordering rows by multiple keys.
Detailed Concept Explanation
Sorting reorganizes DataFrame rows based on the values in one or more columns. You can use either sort() or orderBy()—both are identical aliases in Spark.
By default, Spark sorts data in ascending order. To sort in descending order, you must wrap the column reference in desc().
Code Examples
Input Dataset Preview
Below is the student test scores dataset:
| name | score |
|---|---|
| Alice | 85 |
| Bob | 95 |
| Charlie | 90 |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("Sorting").getOrCreate()
data = [("Alice", 85), ("Bob", 95), ("Charlie", 90)]
df = spark.createDataFrame(data, ["name", "score"])
# Order by score descending
sorted_df = df.orderBy(col("score").desc())
sorted_df.show()
Expected Output
+-------+-----+
| name|score|
+-------+-----+
| Bob| 95|
|Charlie| 90|
| Alice| 85|
+-------+-----+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("SortScala").getOrCreate()
import spark.implicits._
val df = Seq(("Alice", 85), ("Bob", 95), ("Charlie", 90)).toDF("name", "score")
val sorted = df.orderBy($"score".desc)
sorted.show()
Expected Output
+-------+-----+
| name|score|
+-------+-----+
| Bob| 95|
|Charlie| 90|
| Alice| 85|
+-------+-----+
SQL Implementation
SELECT name, score
FROM students
ORDER BY score DESC;
Expected Output
Executing this SQL query returns:
| name | score |
|---|---|
| Bob | 95 |
| Charlie | 90 |
| Alice | 85 |
Common Mistakes
- Assuming Sorting is Low-Cost: Sorting requires data to be compared across all partitions, which triggers a wide dependency shuffle. Avoid sorting intermediate datasets in your pipeline; only sort at the very end when displaying or writing final results.
Best Practices
- Sort at the End: Only apply sorting at the end of your pipeline before writing or outputting data, saving unnecessary shuffles.
Interview Perspective
What happens under the hood when you sort a DataFrame in Spark?
Sorting is a wide transformation. Because values must be compared across the entire dataset, Spark has to shuffle data across executors over the network to arrange the rows in order, which is resource-intensive.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.