Spark SQL Reference

Run declarative SQL queries directly on temporary views or tables.

createOrReplaceTempView

Return: None

Creates a local temporary view with this DataFrame.

Syntax signature:df.createOrReplaceTempView(name)

Code snippet:

python

df.createOrReplaceTempView('users')

Remember: Temp views are session-scoped and will disappear once the active SparkSession terminates.

sql

Return: DataFrame

Executes a SQL query, returning the result as a DataFrame.

Syntax signature:spark.sql(sqlQuery)

Code snippet:

python

spark.sql('SELECT name, salary FROM users WHERE age > 30')

Remember: Spark SQL statements are parsed, optimized, and execute with identical speed to native DataFrame API calls.