scala - 如何将列从十六进制字符串转换为长字符串？

Question

我有一个带有 Icao 列的 DataFrame，其中包含十六进制代码，我想将其转换为 Long 数据类型。我如何在 Spark SQL 中执行此操作？

|  Icao|count|
+------+-----+
|471F8D|81350|
|471F58|79634|
|471F56|79112|
|471F86|78177|
|471F8B|75300|
|47340D|75293|
|471F83|74864|
|471F57|73815|
|471F4A|72290|
|471F5F|72133|
|40612C|69676|

Answer 1

TL;DR 使用 conv标准函数。

conv(num: Column, fromBase: Int, toBase: Int): Column Convert a number in a string column from one base to another.

使用conv，解决方案如下:

scala> icao.show
+------+-----+
|  Icao|count|
+------+-----+
|471F8D|81350|
|471F58|79634|
|471F56|79112|
|471F86|78177|
|471F8B|75300|
|47340D|75293|
|471F83|74864|
|471F57|73815|
|471F4A|72290|
|471F5F|72133|
|40612C|69676|
+------+-----+

// conv is not available by default unless you're in spark-shell
import org.apache.spark.sql.functions.conv

val s1 = icao.withColumn("conv", conv($"Icao", 16, 10))
scala> s1.show
+------+-----+-------+
|  Icao|count|   conv|
+------+-----+-------+
|471F8D|81350|4661133|
|471F58|79634|4661080|
|471F56|79112|4661078|
|471F86|78177|4661126|
|471F8B|75300|4661131|
|47340D|75293|4666381|
|471F83|74864|4661123|
|471F57|73815|4661079|
|471F4A|72290|4661066|
|471F5F|72133|4661087|
|40612C|69676|4219180|
+------+-----+-------+

conv 具有为您提供输入列类型的结果的功能，因此我从字符串开始并获得了字符串。

scala> s1.printSchema
root
 |-- Icao: string (nullable = true)
 |-- count: string (nullable = true)
 |-- conv: string (nullable = true)

如果我使用整数，我就会得到整数。

您可以使用另一种内置方法 cast 转换 conv 的结果(或从输入列的适当类型开始)。

val s2 = icao.withColumn("conv", conv($"Icao", 16, 10) cast "long")
scala> s2.printSchema
root
 |-- Icao: string (nullable = true)
 |-- count: string (nullable = true)
 |-- conv: long (nullable = true)

scala> s2.show
+------+-----+-------+
|  Icao|count|   conv|
+------+-----+-------+
|471F8D|81350|4661133|
|471F58|79634|4661080|
|471F56|79112|4661078|
|471F86|78177|4661126|
|471F8B|75300|4661131|
|47340D|75293|4666381|
|471F83|74864|4661123|
|471F57|73815|4661079|
|471F4A|72290|4661066|
|471F5F|72133|4661087|
|40612C|69676|4219180|
+------+-----+-------+