intermediate
BigQuery
10 min readLast updated: 2026-07-08
Overview
Connect Spark to Google Cloud BigQuery using the BigQuery connector to read and write large-scale data warehouse tables.
What You Will Learn
In this lesson, you will learn:
- BigQuery Connector: Loading the BigQuery connector dependency.
- Authentication: Configuring GCP credentials.
- DataFrame Operations: Reading and writing BigQuery datasets.
Detailed Concept Explanation
To connect Spark to BigQuery, you use Google Cloud's BigQuery connector for Spark. The connector writes intermediate data to a Google Cloud Storage (GCS) temporary bucket before loading it into BigQuery.
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("BigQueryTest") \
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.24.2") \
.getOrCreate()
# Read from a BigQuery table
bq_df = spark.read.format("bigquery") \
.option("table", "my-project.my_dataset.my_table") \
.load()
bq_df.show()
Expected Output
text
+---+-----+
| id| name|
+---+-----+
| 1|Alice|
+---+-----+
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
read.format(bigquery)
option(table)
load()
show()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("BigQueryScala").getOrCreate()
val bqDF = spark.read.format("bigquery")
.option("table", "my-project.my_dataset.my_table")
.load()
bqDF.show()
Expected Output
text
+---+-----+
| id| name|
+---+-----+
| 1|Alice|
+---+-----+
Common Mistakes
- Forgetting Temporary GCS Bucket: Forgetting to configure a temporary GCS bucket when writing data to BigQuery. The connector requires a staging bucket to store intermediate files. Set the option
.option("temporaryGcsBucket", "my-temp-bucket").