intermediate
BigQuery
Overview
Connect Spark to Google Cloud BigQuery using the BigQuery connector to read and write large-scale data warehouse tables.
What You Will Learn
In this lesson, you will learn:
- BigQuery Connector: Loading the BigQuery connector dependency.
- Authentication: Configuring GCP credentials.
- DataFrame Operations: Reading and writing BigQuery datasets.
Detailed Concept Explanation
To connect Spark to BigQuery, you use Google Cloud's BigQuery connector for Spark. The connector writes intermediate data to a Google Cloud Storage (GCS) temporary bucket before loading it into BigQuery.
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("BigQueryTest") \
.config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.24.2") \
.getOrCreate()
# Read from a BigQuery table
bq_df = spark.read.format("bigquery") \
.option("table", "my-project.my_dataset.my_table") \
.load()
bq_df.show()
Expected Output
text
+---+-----+
| id| name|
+---+-----+
| 1|Alice|
+---+-----+
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
read.format(bigquery)
option(table)
load()
show()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("BigQueryScala").getOrCreate()
val bqDF = spark.read.format("bigquery")
.option("table", "my-project.my_dataset.my_table")
.load()
bqDF.show()
Expected Output
text
+---+-----+
| id| name|
+---+-----+
| 1|Alice|
+---+-----+
Common Mistakes
- Forgetting Temporary GCS Bucket: Forgetting to configure a temporary GCS bucket when writing data to BigQuery. The connector requires a staging bucket to store intermediate files. Set the option
.option("temporaryGcsBucket", "my-temp-bucket").
Related Topics
DPA
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.