Union

Overview

Learn how to combine rows from two DataFrames into a single dataset using the union() and unionByName() transformations.

What You Will Learn

Detailed Concept Explanation

Spark provides two methods to combine the rows of two DataFrames:

• df1.union(df2): Combines rows based on column position. The columns in both DataFrames must match in number and data type order.
• df1.unionByName(df2): Combines rows based on column name. This is safer because it automatically resolves column order mismatches between the two DataFrames.

Code Examples

Input Dataset Preview

Below are the transaction logs from different regions:

Logs Region A:

Logs Region B:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Union").getOrCreate()
logs_a = spark.createDataFrame([("Alice", "Login")], ["user", "event"])
logs_b = spark.createDataFrame([("Bob", "Logout")], ["user", "event"])

# Combine the rows of both DataFrames
all_logs = logs_a.union(logs_b)
all_logs.show()

Expected Output

+-----+------+
| user| event|
+-----+------+
|Alice| Login|
|  Bob|Logout|
+-----+------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("UnionScala").getOrCreate()
import spark.implicits._

val logsA = Seq(("Alice", "Login")).toDF("user", "event")
val logsB = Seq(("Bob", "Logout")).toDF("user", "event")

val allLogs = logsA.union(logsB)
allLogs.show()

Expected Output

+-----+------+
| user| event|
+-----+------+
|Alice| Login|
|  Bob|Logout|
+-----+------+

SQL Implementation

SELECT user, event 
FROM logs_a
UNION ALL
SELECT user, event 
FROM logs_b;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Column Order Mismatches with union(): Using .union() on two DataFrames that have the same columns but in a different order. Spark aligns columns by index, not name, which will corrupt your output data. Use .unionByName() instead.

Best Practices

• Use unionByName for Safety: When combining datasets from different sources, use unionByName() to ensure columns are matched correctly by name rather than index.

Interview Perspective

Interactive Challenges

Related Topics

• Joins
• DataFrame Transformations
• Reading Data