beginner

Handling Null Values

By DataPlayArena Technical Review BoardUpdated 2026-07-0810 min readRuntime Verified

Overview

Learn how to locate, fill, and drop missing or null values in a Spark DataFrame to prevent pipeline failures.

What You Will Learn

In this lesson, you will learn:
  • Drop Nulls: Removing rows containing missing values (na.drop()).
  • Fill Nulls: Replacing missing values with defaults (na.fill()).
  • Check Nulls: Locating nulls using isNull().

Detailed Concept Explanation

Missing data (represented as null in Spark) is common in raw data. Spark provides a dedicated .na sub-API on DataFrames to handle missing values:

  • df.na.drop(): Drops rows containing nulls. You can specify whether to drop if any column is null or all columns are null.
  • df.na.fill(value): Replaces null values with a default value.
  • coalesce(*cols): Evaluates columns in order and returns the first non-null value.

Code Examples

Input Dataset Preview

Below is the user registrations dataset containing missing values:

userage
Alice25
Bobnull
nullnull

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("HandlingNulls").getOrCreate()
data = [("Alice", 25), ("Bob", None), (None, None)]
df = spark.createDataFrame(data, ["user", "age"])

# Fill null age with 0 and drop rows where user is null
cleaned_df = df.na.fill({"age": 0}).na.drop(subset=["user"])
cleaned_df.show()

Expected Output

text
+----+---+
|user|age|
+----+---+
|Alice| 25|
| Bob|  0|
+----+---+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
createDataFrame
na.fill(age -> 0)
na.drop(subset=[user])
show()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("NullsScala").getOrCreate()
import spark.implicits._

val df = Seq(
  ("Alice", Some(25)),
  ("Bob", None),
  (null.asInstanceOf[String], None)
).toDF("user", "age")

val cleaned = df.na.fill(Map("age" -> 0)).na.drop(Array("user"))
cleaned.show()

Expected Output

text
+----+---+
|user|age|
+----+---+
|Alice| 25|
| Bob|  0|
+----+---+

SQL Implementation

sql
SELECT 
    COALESCE(user, 'Anonymous') AS user, 
    COALESCE(age, 0) AS age 
FROM registrations
WHERE user IS NOT NULL;

Expected Output

Executing this SQL query returns:

userage
Alice25
Bob0

Common Mistakes

  • Type Mismatch in Fill: Trying to fill an integer column with a string default (e.g. df.na.fill("N/A")). Spark will ignore the replacement unless the column matches the string type. Use a map to fill columns with type-matching defaults.

Best Practices

  • Specify Subset: When dropping nulls, always specify a subset of key columns (e.g. .na.drop(subset=["id"])) instead of dropping the entire row if any column is null.

Interview Perspective

How do you handle null values inside specific columns of a Spark DataFrame?

Use the .na sub-API. To replace nulls with default values, use df.na.fill(value, subset). To drop rows containing nulls, use df.na.drop(subset). For column-level evaluation fallback, use the coalesce function.


Interactive Challenges

Challenge 1: Drop Specific Nulls (Beginner)

Write a PySpark expression to drop any row where the 'email' column contains a null value.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.