python-3.x - Pyspark列: convert string to datetype

Question

我正在尝试将字符串类型的 pyspark 列转换为日期类型，如下所示。

**Date**
31 Mar 2020
2  Apr 2020
29 Jan 2019
8  Sep 2109

所需输出:

31-03-2020
02-04-2020
29-01-2019
08-04-2109  

谢谢。

Answer 1

您可以在内置函数中使用dayofmonth,year,month(或)date_format()(或)from_unixtime(unix_timestamp())对于本例。

示例:

#sample data
df=spark.createDataFrame([("31 Mar 2020",),("2 Apr 2020",),("29 Jan 2019",)],["Date"])
#DataFrame[Date: string]
df.show()
#+-----------+
#|       Date|
#+-----------+
#|31 Mar 2020|
#| 2 Apr 2020|
#|29 Jan 2019|
#+-----------+

from pyspark.sql.functions import *

df.withColumn("new_dt", to_date(col("Date"),"dd MMM yyyy")).\
withColumn("year",year(col("new_dt"))).\
withColumn("month",month(col("new_dt"))).\
withColumn("day",dayofmonth(col("new_dt"))).\
show()

#+-----------+----------+----+-----+---+
#|       Date|    new_dt|year|month|day|
#+-----------+----------+----+-----+---+
#|31 Mar 2020|2020-03-31|2020|    3| 31|
#| 2 Apr 2020|2020-04-02|2020|    4|  2|
#|29 Jan 2019|2019-01-29|2019|    1| 29|
#+-----------+----------+----+-----+---+

#using date_format
df.withColumn("new_dt", to_date(col("Date"),"dd MMM yyyy")).\
withColumn("year",date_format(col("new_dt"),"yyyy")).\
withColumn("month",date_format(col("new_dt"),"MM")).\
withColumn("day",date_format(col("new_dt"),"dd")).show()

#+-----------+----------+----+-----+---+
#|       Date|    new_dt|year|month|day|
#+-----------+----------+----+-----+---+
#|31 Mar 2020|2020-03-31|2020|   03| 31|
#| 2 Apr 2020|2020-04-02|2020|   04| 02|
#|29 Jan 2019|2019-01-29|2019|   01| 29|
#+-----------+----------+----+-----+---+