Working with Columns

Overview

Learn how to create new columns, modify existing ones, and rename column headers in a Spark DataFrame.

What You Will Learn

Detailed Concept Explanation

In Spark, you manipulate DataFrame columns using:

• withColumn(colName, colExpression): Adds a new column or replaces an existing one if colName matches an existing header.
• withColumnRenamed(existingName, newName): Renames a column.

Because DataFrames are immutable, calling these methods returns a new DataFrame rather than modifying the existing one in-place.

Code Examples

Input Dataset Preview

Below is the product prices dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("Columns").getOrCreate()
data = [("Shoes", 100), ("Hat", 25)]
df = spark.createDataFrame(data, ["product", "price_usd"])

# Add tax and rename price_usd to base_price
updated_df = df.withColumn("tax", col("price_usd") * 0.1) \
               .withColumnRenamed("price_usd", "base_price")
updated_df.show()

Expected Output

+-------+----------+----+
|product|base_price| tax|
+-------+----------+----+
|  Shoes|       100|10.0|
|    Hat|        25| 2.5|
+-------+----------+----+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ColumnsScala").getOrCreate()
import spark.implicits._

val df = Seq(("Shoes", 100), ("Hat", 25)).toDF("product", "price_usd")
val updated = df.withColumn("tax", $"price_usd" * 0.1)
  .withColumnRenamed("price_usd", "base_price")
updated.show()

Expected Output

+-------+----------+----+
|product|base_price| tax|
+-------+----------+----+
|  Shoes|       100|10.0|
|    Hat|        25| 2.5|
+-------+----------+----+

SQL Implementation

SELECT product, price_usd AS base_price, price_usd * 0.1 AS tax 
FROM catalog;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Typo in Renamed Target: Passing a column name to withColumnRenamed() that doesn't exist. Spark does not throw an error; it simply leaves the DataFrame unchanged.

Best Practices

• Avoid Chaining excessive withColumn calls: Calling .withColumn() 50 times in a row creates a massive, deep logical plan that can cause Catalyst compilation to exceed memory limits. Group selections inside a single .select() instead.

Interview Perspective

Interactive Challenges

Related Topics

• Selecting Columns
• Handling Null Values
• Built-in Functions