Filtering Data
Overview
Learn how to filter rows in a Spark DataFrame using conditions and logical operators.
What You Will Learn
In this lesson, you will learn:
- Filter API: Using
filter()andwhere(). - Conditions: Combining filters with logical AND/OR operators.
- String Matching: Performing pattern matches like
like()andcontains().
Detailed Concept Explanation
Filtering rows allows you to narrow down your dataset based on one or more conditions. In Spark, filter() and where() are identical aliases—you can use whichever you prefer.
Conditions are evaluated for each row, and only rows that return True are kept.
Code Examples
Input Dataset Preview
Below is the weather sensor readings dataset:
| city | temp |
|---|---|
| New York | 22 |
| Phoenix | 42 |
| Miami | 31 |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("Filtering").getOrCreate()
data = [("New York", 22), ("Phoenix", 42), ("Miami", 31)]
df = spark.createDataFrame(data, ["city", "temp"])
# Filter for temp greater than 30
hot_cities = df.filter(col("temp") > 30)
hot_cities.show()
Expected Output
+-------+----+
| city|temp|
+-------+----+
|Phoenix| 42|
| Miami| 31|
+-------+----+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("FilterScala").getOrCreate()
import spark.implicits._
val df = Seq(("New York", 22), ("Phoenix", 42), ("Miami", 31)).toDF("city", "temp")
val hotCities = df.filter($"temp" > 30)
hotCities.show()
Expected Output
+-------+----+
| city|temp|
+-------+----+
|Phoenix| 42|
| Miami| 31|
+-------+----+
SQL Implementation
SELECT city, temp
FROM readings
WHERE temp > 30;
Expected Output
Executing this SQL query returns:
| city | temp |
|---|---|
| Phoenix | 42 |
| Miami | 31 |
Common Mistakes
- Using Python's logical 'and' or 'or': When combining multiple conditions in PySpark, you must use bitwise operators
&(AND) and|(OR) instead of the Python keywordsandandor. Additionally, wrap each condition in parentheses:(col("temp") > 30) & (col("city") != "Miami").
Best Practices
- Filter Early: Apply your filters as early as possible in your execution flow to minimize the amount of data processed in subsequent steps.
Interview Perspective
What is the difference between filter() and where() in Spark DataFrames?
There is no difference. where() is simply a wrapper function that calls filter(). Both perform identical row-filtering operations and build the same execution plan.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.