Window Functions Reference

Apply rank, row metrics, and offsets over partitioned groups.

Window.partitionBy

Return: WindowSpec

Creates a window boundary grouped by partitioning columns.

Syntax signature:Window.partitionBy(*cols).orderBy(*orderCols)

Code snippet:

python

Window.partitionBy('dept').orderBy('salary')

Remember: Always order partitions to get deterministic rankings or offset values (like dense_rank or lag).

row_number

Return: Column

Assigns a sequential integer starting from 1 to each row inside a window partition.

Syntax signature:row_number()

Code snippet:

python

df.withColumn('rn', row_number().over(windowSpec))

Remember: Useful for deduplication (filtering for rn == 1) to isolate priority transaction records.

dense_rank

Return: Column

Assigns ranks to rows inside a window partition without gaps in ranking values.

Syntax signature:dense_rank()

Code snippet:

python

df.withColumn('dr', dense_rank().over(windowSpec))

Remember: Unlike rank(), dense_rank() ensures there are no numeric gaps if duplicate values share ranks.

lag

Return: Column

Returns the value of a column at a given offset lookback index.

Syntax signature:lag(colName, offset=1)

Code snippet:

python

df.withColumn('prev_sal', lag('salary', 1).over(windowSpec))

Remember: Perfect for calculating periods growth trends (e.g. comparing current row value against previous row value).