Google Cloud Dataproc Reference

Deploy and optimize PySpark workloads on managed GCP Google Dataproc Hadoop clusters.

Dataproc Submit

Return: CLI Command

Submits a PySpark job to a running Dataproc cluster.

Syntax signature:gcloud dataproc jobs submit pyspark

Code snippet:

python

gcloud dataproc jobs submit pyspark app.py --cluster=prod-etl --region=us-central1

Remember: Package dependencies in a wheel file and reference using --py-files flag.

GCS Connector

Return: Storage URI

Reads and writes files directly from Google Cloud Storage buckets.

Syntax signature:gs://bucket-name/path

Code snippet:

python

df = spark.read.parquet('gs://my-data-lake/raw/sales/')

Remember: Dataproc includes the GCS connector natively. Ensure service accounts have read/write IAM storage permissions.