我在 python/pyspark 中有一个数据框。这些列具有特殊字符,例如点(.)、空格、方括号(()) 和圆括号{}。以他们的名字命名。
现在我想重命名列名称,如果有点和空格,则用下划线替换它们,如果有 () 和 {},则将它们从列名称中删除。
我已经做到了
df1 = df.toDF(*(re.sub(r'[\.\s]+', '_', c) for c in df.columns))
这样,我就可以用 Unable to do the second bit i.e if () and {} are there 只是将它们从列名称中删除,用下划线替换点和空格。
我们如何实现这一目标。
最佳答案
如果您有 pyspark 数据框,您可以尝试使用 withColumnRenamed 函数来重命名列。我确实以我的方式尝试过,看看并根据您的更改进行自定义。
>>> l=[('some value1','some value2','some value 3'),('some value4','some value5','some value 6')]
>>> l_schema = StructType([StructField("col1.some valwith(in)and{around}",StringType(),True),StructField("col2.some valwith()and{}",StringType(),True),StructField("col3 some()valwith.and{}",StringType(),True)])
>>> reps=('.','_'),(' ','_'),('(',''),(')',''),('{','')('}','')
>>> rdd = sc.parallelize(l)
>>> df = sqlContext.createDataFrame(rdd,l_schema)
>>> df.printSchema()
root
|-- col1.some valwith(in)and{around}: string (nullable = true)
|-- col2.some valwith()and{}: string (nullable = true)
|-- col3 some()valwith.and{}: string (nullable = true)
>>> df.show()
+------------------------+------------------------+------------------------+
|col1.some valwith(in)and{around}|col2.some valwith()and{}|col3 some()valwith.and{}|
+------------------------+------------------------+------------------------+
| some value1| some value2| some value 3|
| some value4| some value5| some value 6|
+------------------------+------------------------+------------------------+
>>> def colrename(x):
... return reduce(lambda a,kv : a.replace(*kv),reps,x)
>>> for i in df.schema.names:
... df = df.withColumnRenamed(i,colrename(i))
>>> df.printSchema()
root
|-- col1_some_valwithinandaround: string (nullable = true)
|-- col2_some_valwithand: string (nullable = true)
|-- col3_somevalwith_and: string (nullable = true)
>>> df.show()
+--------------------+--------------------+--------------------+
|col1_some_valwithinandaround|col2_some_valwithand|col3_somevalwith_and|
+--------------------+--------------------+--------------------+
| some value1| some value2| some value 3|
| some value4| some value5| some value 6|
+--------------------+--------------------+--------------------+
关于python - 在 python 或 Pyspark 数据框中重命名具有特殊字符的列,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/42753559/