pyspark.sql.functions.transform_values

pyspark.sql.functions.transform_values(col: ColumnOrName, f: Callable[[pyspark.sql.column.Column, pyspark.sql.column.Column], pyspark.sql.column.Column]) → pyspark.sql.column.Column

Applies a function to every key-value pair in a map and returns a map with the results of those applications as the new values for the pairs.

New in version 3.1.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

colColumn or str

name of column or expression

ffunction

a binary function (k: Column, v: Column) -> Column... Can use methods of Column, functions defined in pyspark.sql.functions and Scala UserDefinedFunctions. Python UserDefinedFunctions are not supported (SPARK-27052).

Returns

Column

a new map of enties where new values were calculated by applying given function to each key value argument.

Examples

>>> df = spark.createDataFrame([(1, {"IT": 10.0, "SALES": 2.0, "OPS": 24.0})], ("id", "data"))
>>> row = df.select(transform_values(
...     "data", lambda k, v: when(k.isin("IT", "OPS"), v + 10.0).otherwise(v)
... ).alias("new_data")).head()
>>> sorted(row["new_data"].items())
[('IT', 20.0), ('OPS', 34.0), ('SALES', 2.0)]

previous

pyspark.sql.functions.transform_keys

next

pyspark.sql.functions.map_filter