scala - 如何将自定义日期时间格式转换为时间戳？

Question

知道为什么我会得到下面的结果吗？

scala> val b = to_timestamp($"DATETIME", "ddMMMYYYY:HH:mm:ss")
b: org.apache.spark.sql.Column = to_timestamp(`DATETIME`, 'ddMMMYYYY:HH:mm:ss')

scala> sourceRawData.withColumn("ts", b).show(6,false)
+------------------+-------------------+-----------+--------+----------------+---------+-------------------+
|DATETIME          |LOAD_DATETIME      |SOURCE_BANK|EMP_NAME|HEADER_ROW_COUNT|EMP_HOURS|ts                 |
+------------------+-------------------+-----------+--------+----------------+---------+-------------------+
|01JAN2017:01:02:03|01JAN2017:01:02:03 | RBS       | Naveen |100             |15.23    |2017-01-01 01:02:03|
|15MAR2017:01:02:03|15MAR2017:01:02:03 | RBS       | Naveen |100             |115.78   |2017-01-01 01:02:03|
|02APR2015:23:24:25|02APR2015:23:24:25 | RBS       |Arun    |200             |2.09     |2014-12-28 23:24:25|
|28MAY2010:12:13:14| 28MAY2010:12:13:14|RBS        |Arun    |100             |30.98    |2009-12-27 12:13:14|
|04JUN2018:10:11:12|04JUN2018:10:11:12 |XZX        | Arun   |400             |12.0     |2017-12-31 10:11:12|
+------------------+-------------------+-----------+--------+----------------+---------+-------------------+

我正在尝试将 DATETIME(采用 ddMMMYY:HH:mm:ss 格式)转换为时间戳(显示在上面最后一列中)，但它似乎没有转换为正确的值。 我引用了以下帖子但没有帮助:

Better way to convert a string field into timestamp in Spark

谁能帮帮我？

Answer 1

使用 y(年)而不是 Y(星期年):

spark.sql("SELECT to_timestamp('04JUN2018:10:11:12', 'ddMMMyyyy:HH:mm:ss')").show
// +--------------------------------------------------------+
// |to_timestamp('04JUN2018:10:11:12', 'ddMMMyyyy:HH:mm:ss')|
// +--------------------------------------------------------+
// |                                     2018-06-04 10:11:12|
// +--------------------------------------------------------+

另一个例子:

scala> sql("select to_timestamp('12/08/2020 1:24:21 AM', 'MM/dd/yyyy H:mm:ss a')").show
+-------------------------------------------------------------+
|to_timestamp('12/08/2020 1:24:21 AM', 'MM/dd/yyyy H:mm:ss a')|
+-------------------------------------------------------------+
|                                          2020-12-08 01:24:21|
+-------------------------------------------------------------+