Temporary Views
Overview
Learn how to register a Spark DataFrame as a temporary view, allowing you to query it using standard SQL statements.
What You Will Learn
In this lesson, you will learn:
- Create Views: Registering local views using
createOrReplaceTempView(). - SQL Execution: Running queries on top of views via
spark.sql(). - Scope Limits: Understanding the lifetime boundaries of temporary vs global views.
Detailed Concept Explanation
If you prefer writing SQL queries over DataFrame transformations, you can register your DataFrame as a temporary view:
df.createOrReplaceTempView("view_name")
Once registered, you can query this view using standard SQL via the SparkSession:
spark.sql("SELECT * FROM view_name WHERE price > 500")
Scope Boundaries
- Temporary Views: Scoped to the active
SparkSession. They are automatically deleted when the session terminates. - Global Temporary Views: Scoped to the entire Spark application, making them accessible across different SparkSessions. They are registered under the system namespace
global_temp.
Code Examples
Input Dataset Preview
Below is the flight bookings dataset:
| flight | destination |
|---|---|
| AA101 | London |
| UA204 | Paris |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("TempViews").getOrCreate()
data = [("AA101", "London"), ("UA204", "Paris")]
df = spark.createDataFrame(data, ["flight", "destination"])
# Register the DataFrame as a temporary view
df.createOrReplaceTempView("bookings")
# Query the view using standard SQL
result_df = spark.sql("SELECT * FROM bookings WHERE destination = 'London'")
result_df.show()
Expected Output
+------+-----------+
|flight|destination|
+------+-----------+
| AA101| London|
+------+-----------+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("TempViewsScala").getOrCreate()
import spark.implicits._
val df = Seq(("AA101", "London"), ("UA204", "Paris")).toDF("flight", "destination")
df.createOrReplaceTempView("bookings")
val result = spark.sql("SELECT * FROM bookings WHERE destination = 'London'")
result.show()
Expected Output
+------+-----------+
|flight|destination|
+------+-----------+
| AA101| London|
+------+-----------+
SQL Implementation
-- Querying the registered temporary view directly in SQL
SELECT flight
FROM bookings
WHERE destination = 'Paris';
Expected Output
Executing this SQL query returns:
| flight |
|---|
| UA204 |
Common Mistakes
- Accessing Views After Session Close: Trying to query a temporary view after calling
spark.stop(). Since temporary views are bound to the active session, they are deleted when the session stops.
Best Practices
- Use createOrReplace: Always use
createOrReplaceTempView()instead ofcreateTempView(). This prevents crashes if your code runs multiple times and attempts to register the same view name.
Interview Perspective
What is the difference between a temporary view and a global temporary view in Spark?
A temporary view is bound to the specific SparkSession that created it. A global temporary view is shared across all sessions within the Spark application. Global views are kept in the system database global_temp and must be queried using their fully qualified name: SELECT * FROM global_temp.view_name.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.