我需要从现有的 DataFrame 创建一个 DataFrame,我还需要在其中更改架构。
我有一个像这样的 DataFrame:
+-----------+----------+-------------+
|Id |Position |playerName |
+-----------+-----------+------------+
|10125 |Forward |Messi |
|10126 |Forward |Ronaldo |
|10127 |Midfield |Xavi |
|10128 |Midfield |Neymar |
我是使用下面给出的案例类创建的:
case class caseClass (
Id: Int = "",
Position : String = "" ,
playerName : String = ""
)
现在我需要在 Struct 类型下创建 Playername 和 position。
即,
我需要创建另一个带有模式的 DataFrame,
root
|-- Id: int (nullable = true)
|-- playerDetails: struct (nullable = true)
| |--playername: string (nullable = true)
| |--Position: string (nullable = true)
我通过引用链接做了以下代码来创建一个新的数据框 https://medium.com/@mrpowers/adding-structtype-columns-to-spark-dataframes-b44125409803
我的模式是
List(
StructField("Id", IntegerType, true),
StructField("Position",StringType, true),
StructField("playerName", StringType,true)
)
我试过下面的代码
spark.sparkContext.parallelize(data),
myschema
)
但我做不到。
我看到了类似的问题 Change schema of existing dataframe但我无法理解解决方案。
有没有直接在case类中实现StructType的解决方案?所以我认为我不需要创建自己的架构来创建结构类型值。
最佳答案
可以使用函数“struct”:
// data
val playersDF = Seq(
(10125, "Forward", "Messi"),
(10126, "Forward", "Ronaldo"),
(10127, "Midfield", "Xavi"),
(10128, "Midfield", "Neymar")
).toDF("Id", "Position", "playerName")
// action
val playersStructuredDF = playersDF.select($"Id", struct("playerName", "Position").as("playerDetails"))
// display
playersStructuredDF.printSchema()
playersStructuredDF.show(false)
输出:
root
|-- Id: integer (nullable = false)
|-- playerDetails: struct (nullable = false)
| |-- playerName: string (nullable = true)
| |-- Position: string (nullable = true)
+-----+------------------+
|Id |playerDetails |
+-----+------------------+
|10125|[Messi, Forward] |
|10126|[Ronaldo, Forward]|
|10127|[Xavi, Midfield] |
|10128|[Neymar, Midfield]|
+-----+------------------+
关于java - 如何从现有 Dataframe 创建 Dataframe 并将特定字段设为 Struct 类型?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/55471371/