python - 再向模式添加一个 StructField

Question

我的 PySpark 数据框具有以下架构:

schema = spark_df.printSchema()

root
 |-- field_1: double (nullable = true)
 |-- field_2: double (nullable = true)
 |-- field_3 (nullable = true)
 |-- field_4: double (nullable = true)
 |-- field_5: double (nullable = true)
 |-- field_6: double (nullable = true)

我想再向架构中添加一个 StructField，因此新架构如下所示:

root
 |-- field_1: double (nullable = true)
 |-- field_1: double (nullable = true)
 |-- field_2: double (nullable = true)
 |-- field_3 (nullable = true)
 |-- field_4: double (nullable = true)
 |-- field_5: double (nullable = true)
 |-- field_6: double (nullable = true)

我知道我可以手动创建一个 new_schema，如下所示:

new_schema = StructType([StructField("field_0", StringType(), True),
                            :
                         StructField("field_6", IntegerType(), True)])

这适用于少量字段，但如果我有数百个字段则无法生成。所以我想知道有没有更优雅的方法来在模式的开头添加一个新字段？谢谢!

Answer 1

您可以复制现有字段并保留:

to_prepend = [StructField("field_0", StringType(), True)] 

StructType(to_prepend + df.schema.fields)