Temporary Views

Overview

Learn how to register a Spark DataFrame as a temporary view, allowing you to query it using standard SQL statements.

What You Will Learn

Detailed Concept Explanation

If you prefer writing SQL queries over DataFrame transformations, you can register your DataFrame as a temporary view: df.createOrReplaceTempView("view_name")

Once registered, you can query this view using standard SQL via the SparkSession: spark.sql("SELECT * FROM view_name WHERE price > 500")

Scope Boundaries

1. Temporary Views: Scoped to the active SparkSession. They are automatically deleted when the session terminates.
2. Global Temporary Views: Scoped to the entire Spark application, making them accessible across different SparkSessions. They are registered under the system namespace global_temp.

Code Examples

Input Dataset Preview

Below is the flight bookings dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("TempViews").getOrCreate()
data = [("AA101", "London"), ("UA204", "Paris")]
df = spark.createDataFrame(data, ["flight", "destination"])

# Register the DataFrame as a temporary view
df.createOrReplaceTempView("bookings")

# Query the view using standard SQL
result_df = spark.sql("SELECT * FROM bookings WHERE destination = 'London'")
result_df.show()

Expected Output

+------+-----------+
|flight|destination|
+------+-----------+
| AA101|     London|
+------+-----------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("TempViewsScala").getOrCreate()
import spark.implicits._

val df = Seq(("AA101", "London"), ("UA204", "Paris")).toDF("flight", "destination")
df.createOrReplaceTempView("bookings")

val result = spark.sql("SELECT * FROM bookings WHERE destination = 'London'")
result.show()

Expected Output

+------+-----------+
|flight|destination|
+------+-----------+
| AA101|     London|
+------+-----------+

SQL Implementation

-- Querying the registered temporary view directly in SQL
SELECT flight 
FROM bookings 
WHERE destination = 'Paris';

Expected Output

Executing this SQL query returns:

Common Mistakes

• Accessing Views After Session Close: Trying to query a temporary view after calling spark.stop(). Since temporary views are bound to the active session, they are deleted when the session stops.

Best Practices

• Use createOrReplace: Always use createOrReplaceTempView() instead of createTempView(). This prevents crashes if your code runs multiple times and attempts to register the same view name.

Interview Perspective

Interactive Challenges

Related Topics

• Spark SQL Basics
• SparkSession Lifecycle
• Reading Data