String Functions Reference

Trim, concatenate, capitalize, and extract substrings from text data.

concat_ws

Return: Column

Concatenates multiple string columns using a specified separator symbol.

Syntax signature:concat_ws(sep, *cols)

Code snippet:

python

concat_ws(' ', col('first'), col('last'))

Remember: Safe function: automatically skips NULL values without crashing or producing NULL output strings.

substring

Return: Column

Extracts a substring starting at pos index with a specified length.

Syntax signature:substring(col, pos, len)

Code snippet:

python

substring(col('zipcode'), 1, 5)

Remember: Spark string position indexing is 1-based. Negative pos starts counting from the end of the string.

trim

Return: Column

Removes leading and trailing space characters from text columns.

Syntax signature:trim(col)

Code snippet:

python

trim(col('input'))

Remember: Always trim incoming keys prior to joining to prevent mismatches caused by trailing spaces.