apache-spark - Spark SQL - 处理列名中的双引号

Question

我有一个列名类似于的 HIVE 表:column_"COLUMN_NAME"
我的原始查询如下。

spark.sql("SELECT from_unixtime(unix_timestamp(substr(time, 1, 23), 'dd.MM.yyyy HH:mm:ss.SSS')) AS timestamp, cast(regexp_replace(column_"COLUMN_NAME"",',','.') as float) AS Column FROM table")

这显然行不通。我尝试了各种方法来转义列名中的引号，但反斜杠和反引号都不能解决问题。

有任何想法吗？

Answer 1

您在这里有两个选项，但在这两种情况下，您都需要将包含双引号的列名括在反引号中。

首先创建一个例子

data = [
    ('01.01.2019 12:34:56.78910', '123,456')
]
df = spark.createDataFrame(data, ['time', 'column_"COLUMN_NAME"'])
df.show()
#+-------------------------+--------------------+
#|time                     |column_"COLUMN_NAME"|
#+-------------------------+--------------------+
#|01.01.2019 12:34:56.78910|123,456             |
#+-------------------------+--------------------+

# register this as a temp table
df.createOrReplaceTempView("table")

选项 1:使用三重引号:

query = """SELECT 
from_unixtime(unix_timestamp(substr(time, 1, 23), 'dd.MM.yyyy HH:mm:ss.SSS')) AS timestamp, 
cast(regexp_replace(`column_"COLUMN_NAME"`,',','.') as float) AS Column 
FROM table"""
spark.sql(query).show()
#+-------------------+-------+
#|          timestamp| Column|
#+-------------------+-------+
#|2019-01-01 12:34:56|123.456|
#+-------------------+-------+

选项 2:转义双引号

query = "SELECT from_unixtime(unix_timestamp(substr(time, 1, 23), 'dd.MM.yyyy HH:mm:ss.SSS')) AS timestamp, cast(regexp_replace(`column_\"COLUMN_NAME\"`,',','.') as float) AS Column FROM table"
spark.sql(query).show()
#Same as above