访问数据框 Row
值时如何处理 Null? Null pointer Exception 真的需要手动处理吗?必须有更好的解决方案。
case class FirstThing(id:Int, thing:String, other:Option[Double])
val df = Seq(FirstThing(1, "first", None), FirstThing(1, "second", Some(2)), FirstThing(1, "third", Some(3))).toDS
df.show
val list = df.groupBy("id").agg(collect_list(struct("thing", "other")).alias("mylist"))
list.show(false)
NPE 失败:
val xxxx = udf((t:Seq[Row])=> t.map(elem => elem.getDouble(1)))
list.withColumn("aa", xxxx(col("mylist"))).show(false)
这奇怪地给出了 0:
val xxxx = udf((t:Seq[Row])=> t.map(elem => elem.getAs[Double]("other")))
list.withColumn("aa", xxxx(col("mylist"))).show(false)
+---+-----------------------------------------+---------------+
|id |mylist |aa |
+---+-----------------------------------------+---------------+
|1 |[[first,null], [second,2.0], [third,3.0]]|[0.0, 2.0, 3.0]|
+---+-----------------------------------------+---------------+
遗憾的是,这种适用于数据框/数据集的方法也失败了:
val xxxx = udf((t:Seq[Row])=> t.map(elem => elem.getAs[Option[Double]]("other")))
list.withColumn("aa", xxxx(col("mylist"))).show(false)
ClassCastException: java.lang.Double cannot be cast to scala.Option
最佳答案
使用 getAs[Double]
并将其包装在 Option
中将给出预期的结果:
val xxxx = udf((t: Seq[Row])=> t.map(elem => Option(elem.getAs[Double]("other"))))
list.withColumn("aa", xxxx($"mylist")).show(false)
+---+-----------------------------------------+----------------+
|id |mylist |aa |
+---+-----------------------------------------+----------------+
|1 |[[first,null], [second,2.0], [third,3.0]]|[null, 2.0, 3.0]|
+---+-----------------------------------------+----------------+
getAs[Option[Double]]
不起作用的原因可能是数据框架构不了解列具有选项。 udf 之前的架构:
root
|-- id: integer (nullable = false)
|-- mylist: array (nullable = true)
| |-- element: struct (containsNull = true)
| | |-- thing: string (nullable = true)
| | |-- other: double (nullable = true)
关于scala - 数据帧空值在 UDF 后转换为 0。为什么?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/52399488/