Built-in Functions

Overview

Learn how to use Spark's built-in SQL functions to transform strings, dates, and numbers efficiently.

What You Will Learn

Detailed Concept Explanation

Spark provides hundreds of built-in functions in the pyspark.sql.functions module. These functions are written in Scala and execute natively inside the JVM, making them highly optimized.

Common categories:

• String: upper(), lower(), concat(), substring(), trim().
• Date/Time: current_date(), to_date(), date_add(), date_sub(), year().
• Math: round(), abs(), sqrt(), ceil(), floor().

Always prefer using these built-in functions over writing custom User-Defined Functions (UDFs). UDFs act as a black box to Spark's Catalyst Optimizer, preventing query optimizations.

Code Examples

Input Dataset Preview

Below is the raw user registrations dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession
from pyspark.sql.functions import trim, upper, year, col

spark = SparkSession.builder.appName("BuiltIns").getOrCreate()
data = [("  alice ", "2026-01-15"), (" bob", "2026-03-20")]
df = spark.createDataFrame(data, ["raw_name", "reg_date"])

# Format string and extract year
cleaned_df = df.withColumn("clean_name", upper(trim(col("raw_name")))) \
               .withColumn("reg_year", year(col("reg_date")))
cleaned_df.show()

Expected Output

+--------+----------+----------+--------+
|raw_name|  reg_date|clean_name|reg_year|
+--------+----------+----------+--------+
|  alice |2026-01-15|     ALICE|    2026|
|     bob|2026-03-20|       BOB|    2026|
+--------+----------+----------+--------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

val spark = SparkSession.builder().appName("BuiltinsScala").getOrCreate()
import spark.implicits._

val df = Seq(("  alice ", "2026-01-15"), (" bob", "2026-03-20")).toDF("raw_name", "reg_date")
val cleaned = df.withColumn("clean_name", upper(trim($"raw_name")))
  .withColumn("reg_year", year($"reg_date"))
cleaned.show()

Expected Output

+--------+----------+----------+--------+
|raw_name|  reg_date|clean_name|reg_year|
+--------+----------+----------+--------+
|  alice |2026-01-15|     ALICE|    2026|
|     bob|2026-03-20|       BOB|    2026|
+--------+----------+----------+--------+

SQL Implementation

SELECT 
    UPPER(TRIM(raw_name)) AS clean_name, 
    YEAR(reg_date) AS reg_year 
FROM users;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Writing Custom Python UDFs for Simple Changes: Writing custom Python UDFs to perform simple operations (like converting a string to uppercase or calculating an absolute value). Python UDFs run outside the JVM, forcing Spark to serialize data between the JVM and Python processes, which is extremely slow. Use Spark's built-in functions instead.

Best Practices

• Prefer Built-ins: Always check pyspark.sql.functions for a built-in solution before writing a custom Python function.

Interview Perspective

Interactive Challenges

Related Topics

• Working with Columns
• DataFrame Transformations
• Handling Null Values