Working with Columns
Overview
Learn how to create new columns, modify existing ones, and rename column headers in a Spark DataFrame.
What You Will Learn
In this lesson, you will learn:
- Add Columns: Appending calculated fields using
withColumn(). - Modify Values: Applying transformations to existing columns.
- Rename Columns: Changing column names using
withColumnRenamed().
Detailed Concept Explanation
In Spark, you manipulate DataFrame columns using:
withColumn(colName, colExpression): Adds a new column or replaces an existing one ifcolNamematches an existing header.withColumnRenamed(existingName, newName): Renames a column.
Because DataFrames are immutable, calling these methods returns a new DataFrame rather than modifying the existing one in-place.
Code Examples
Input Dataset Preview
Below is the product prices dataset:
| product | price_usd |
|---|---|
| Shoes | 100 |
| Hat | 25 |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("Columns").getOrCreate()
data = [("Shoes", 100), ("Hat", 25)]
df = spark.createDataFrame(data, ["product", "price_usd"])
# Add tax and rename price_usd to base_price
updated_df = df.withColumn("tax", col("price_usd") * 0.1) \
.withColumnRenamed("price_usd", "base_price")
updated_df.show()
Expected Output
+-------+----------+----+
|product|base_price| tax|
+-------+----------+----+
| Shoes| 100|10.0|
| Hat| 25| 2.5|
+-------+----------+----+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("ColumnsScala").getOrCreate()
import spark.implicits._
val df = Seq(("Shoes", 100), ("Hat", 25)).toDF("product", "price_usd")
val updated = df.withColumn("tax", $"price_usd" * 0.1)
.withColumnRenamed("price_usd", "base_price")
updated.show()
Expected Output
+-------+----------+----+
|product|base_price| tax|
+-------+----------+----+
| Shoes| 100|10.0|
| Hat| 25| 2.5|
+-------+----------+----+
SQL Implementation
SELECT product, price_usd AS base_price, price_usd * 0.1 AS tax
FROM catalog;
Expected Output
Executing this SQL query returns:
| product | base_price | tax |
|---|---|---|
| Shoes | 100 | 10.0 |
| Hat | 25 | 2.5 |
Common Mistakes
- Typo in Renamed Target: Passing a column name to
withColumnRenamed()that doesn't exist. Spark does not throw an error; it simply leaves the DataFrame unchanged.
Best Practices
- Avoid Chaining excessive withColumn calls: Calling
.withColumn()50 times in a row creates a massive, deep logical plan that can cause Catalyst compilation to exceed memory limits. Group selections inside a single.select()instead.
Interview Perspective
What is the performance drawback of chaining multiple withColumn calls in PySpark?
Chaining multiple withColumn() calls results in Spark building a deep, nested logical plan. This can slow down Catalyst's query plan optimization. For large-scale column transformations, it is better to define them inside a single select() statement.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.