beginner

Working with Columns

By DataPlayArena Technical Review BoardUpdated 2026-07-088 min readRuntime Verified

Overview

Learn how to create new columns, modify existing ones, and rename column headers in a Spark DataFrame.

What You Will Learn

In this lesson, you will learn:
  • Add Columns: Appending calculated fields using withColumn().
  • Modify Values: Applying transformations to existing columns.
  • Rename Columns: Changing column names using withColumnRenamed().

Detailed Concept Explanation

In Spark, you manipulate DataFrame columns using:

  • withColumn(colName, colExpression): Adds a new column or replaces an existing one if colName matches an existing header.
  • withColumnRenamed(existingName, newName): Renames a column.

Because DataFrames are immutable, calling these methods returns a new DataFrame rather than modifying the existing one in-place.


Code Examples

Input Dataset Preview

Below is the product prices dataset:

productprice_usd
Shoes100
Hat25

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("Columns").getOrCreate()
data = [("Shoes", 100), ("Hat", 25)]
df = spark.createDataFrame(data, ["product", "price_usd"])

# Add tax and rename price_usd to base_price
updated_df = df.withColumn("tax", col("price_usd") * 0.1) \
               .withColumnRenamed("price_usd", "base_price")
updated_df.show()

Expected Output

text
+-------+----------+----+
|product|base_price| tax|
+-------+----------+----+
|  Shoes|       100|10.0|
|    Hat|        25| 2.5|
+-------+----------+----+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
createDataFrame
withColumn(tax)
withColumnRenamed(base_price)
show()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ColumnsScala").getOrCreate()
import spark.implicits._

val df = Seq(("Shoes", 100), ("Hat", 25)).toDF("product", "price_usd")
val updated = df.withColumn("tax", $"price_usd" * 0.1)
  .withColumnRenamed("price_usd", "base_price")
updated.show()

Expected Output

text
+-------+----------+----+
|product|base_price| tax|
+-------+----------+----+
|  Shoes|       100|10.0|
|    Hat|        25| 2.5|
+-------+----------+----+

SQL Implementation

sql
SELECT product, price_usd AS base_price, price_usd * 0.1 AS tax 
FROM catalog;

Expected Output

Executing this SQL query returns:

productbase_pricetax
Shoes10010.0
Hat252.5

Common Mistakes

  • Typo in Renamed Target: Passing a column name to withColumnRenamed() that doesn't exist. Spark does not throw an error; it simply leaves the DataFrame unchanged.

Best Practices

  • Avoid Chaining excessive withColumn calls: Calling .withColumn() 50 times in a row creates a massive, deep logical plan that can cause Catalyst compilation to exceed memory limits. Group selections inside a single .select() instead.

Interview Perspective

What is the performance drawback of chaining multiple withColumn calls in PySpark?

Chaining multiple withColumn() calls results in Spark building a deep, nested logical plan. This can slow down Catalyst's query plan optimization. For large-scale column transformations, it is better to define them inside a single select() statement.


Interactive Challenges

Challenge 1: Add Constant Column (Beginner)

How do you add a column named 'country' with a constant value of 'USA' to all rows in a DataFrame?

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.