Array Functions Reference

Extract, sort, distinct, and flatten nested array list column schemas.

array_contains

Return: Column

Checks if a specified value is present inside an array column.

Syntax signature:array_contains(col, value)

Code snippet:

python

array_contains(col('tags'), 'active')

Remember: Returns True if match exists, False if not, or NULL if the array column structure is NULL.

explode

Return: Column

Creates a new row for each element in the given array column.

Syntax signature:explode(col)

Code snippet:

python

df.select(explode(col('fruits')))

Remember: Warning: explode drops parent rows if the array column is empty or NULL. Use explode_outer to preserve them.

array_distinct

Return: Column

Removes duplicate values from the array collection.

Syntax signature:array_distinct(col)

Code snippet:

python

array_distinct(col('numbers'))

Remember: Maintains the original order of remaining distinct elements.