Structured Streaming Reference

Process real-time incremental streams from Kafka or Cloud sources.

readStream

Return: DataFrame

Sets up an active streaming ingest read source schema.

Syntax signature:spark.readStream.format('kafka').load()

Code snippet:

python

spark.readStream.format('kafka').option('subscribe', 'orders').load()

Remember: Loads directories dynamically as streams if parsing file systems (e.g. cloud directories).

writeStream.trigger

Return: DataStreamWriter

Defines triggering frequency constraints on streaming executions.

Syntax signature:df.writeStream.trigger(processingTime='10 seconds')

Code snippet:

python

df.writeStream.trigger(availableNow=True)

Remember: availableNow=True processes incoming datasets dynamically and terminates, acting like a batch ETL loop.