beginner
Filtering Data
8 min readLast updated: 2026-07-08
Overview
Learn how to filter rows in a Spark DataFrame using conditions and logical operators.
What You Will Learn
In this lesson, you will learn:
- Filter API: Using
filter()andwhere(). - Conditions: Combining filters with logical AND/OR operators.
- String Matching: Performing pattern matches like
like()andcontains().
Detailed Concept Explanation
Filtering rows allows you to narrow down your dataset based on one or more conditions. In Spark, filter() and where() are identical aliases—you can use whichever you prefer.
Conditions are evaluated for each row, and only rows that return True are kept.
Code Examples
Input Dataset Preview
Below is the weather sensor readings dataset:
| city | temp |
|---|---|
| New York | 22 |
| Phoenix | 42 |
| Miami | 31 |
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("Filtering").getOrCreate()
data = [("New York", 22), ("Phoenix", 42), ("Miami", 31)]
df = spark.createDataFrame(data, ["city", "temp"])
# Filter for temp greater than 30
hot_cities = df.filter(col("temp") > 30)
hot_cities.show()
Expected Output
text
+-------+----+
| city|temp|
+-------+----+
|Phoenix| 42|
| Miami| 31|
+-------+----+
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
createDataFrame
filter(temp > 30)
show()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("FilterScala").getOrCreate()
import spark.implicits._
val df = Seq(("New York", 22), ("Phoenix", 42), ("Miami", 31)).toDF("city", "temp")
val hotCities = df.filter($"temp" > 30)
hotCities.show()
Expected Output
text
+-------+----+
| city|temp|
+-------+----+
|Phoenix| 42|
| Miami| 31|
+-------+----+
SQL Implementation
sql
SELECT city, temp
FROM readings
WHERE temp > 30;
Expected Output
Executing this SQL query returns:
| city | temp |
|---|---|
| Phoenix | 42 |
| Miami | 31 |
Common Mistakes
- Using Python's logical 'and' or 'or': When combining multiple conditions in PySpark, you must use bitwise operators
&(AND) and|(OR) instead of the Python keywordsandandor. Additionally, wrap each condition in parentheses:(col("temp") > 30) & (col("city") != "Miami").
Best Practices
- Filter Early: Apply your filters as early as possible in your execution flow to minimize the amount of data processed in subsequent steps.
Interview Perspective
What is the difference between filter() and where() in Spark DataFrames?
There is no difference. where() is simply a wrapper function that calls filter(). Both perform identical row-filtering operations and build the same execution plan.