AWS EMR Reference

Configure and run scalable Spark operations on Amazon Web Services Elastic MapReduce.

EMR Step submit

Return: CLI Command

Submits a PySpark job as a step to an EMR cluster.

Syntax signature:aws emr add-steps --cluster-id ...

Code snippet:

python

aws emr add-steps --cluster-id j-123 --steps Type=Spark,Args=[s3://my-code/app.py]

Remember: Steps run sequentially. Choose ActionOnFailure=CONTINUE to run following operations if one falls.

S3A Protocol File Access

Return: Storage URI

Access files stored on AWS Amazon Simple Storage Service.

Syntax signature:s3a://bucket-name/path

Code snippet:

python

df = spark.read.parquet('s3a://prod-data-bucket/raw/logs/')

Remember: Always use s3a:// instead of s3:// or s3n:// for high-throughput multi-part file transfers.