Withcolumn With Udf Yields Attributeerror: 'nonetype' Object Has No Attribute '_jvm'

I am trying to replace some values in a spark dataframe by using a UDF, but keep on getting the same error. While debugging I found out it doesn't really depend on the dataframe I

Solution 1:

You are very close, it is complaining because you cannot use lit within a udf :) lit is used on column level, not on row level.

l = [('Alice', 1)]
df = spark.createDataFrame(l)
df.show()

+-----+---+
|   _1| _2|
+-----+---+
|Alice|  1|
+-----+---+

df = df.withColumn("_1", udf(lambda x: x+x, StringType())("_1"))
# this would produce the same result, but lit is not necessary here
# df = df.withColumn("_1", udf(lambda x: x+x, StringType()(lit(df["_1"])))
df.show()

+----------+---+
|        _1| _2|
+----------+---+
|AliceAlice|  1|
+----------+---+