我有一个 pyspark 数据框,如下面的输入数据。我想将产品名称列中的值拆分为空白。然后我想使用前 3 个值创建新列。我在下面有示例输入和输出数据。有人可以建议如何使用 pyspark 执行此操作吗?
输入数据:
+------+-------------------+
|id |productname |
+------+-------------------+
|235832|EXTREME BERRY Sweet|
|419736|BLUE CHASER SAUCE |
|124513|LAAVA C2L5 |
+------+-------------------+
输出:
+------+-------------------+-------------+-------------+-------------+
|id |productname |product1 |product2 |product3 |
+------+-------------------+-------------+-------------+-------------+
|235832|EXTREME BERRY Sweet|EXTREME |BERRY |Sweet |
|419736|BLUE CHASER SAUCE |BLUE |CHASER |SAUCE |
|124513|LAAVA C2L5 |LAAVA |C2L5 | |
+------+-------------------+-------------+-------------+-------------+
最佳答案
拆分
产品名称列,然后使用element_at
(或)创建新列.getItem()
索引值。
df.withColumn("tmp",split(col("productname"),"\s+")).\
withColumn("product1",element_at(col("tmp"),1)).\
withColumn("product2",element_at(col("tmp"),2)).\
withColumn("product3",coalesce(element_at(col("tmp"),3),lit(""))).drop("tmp").show()
#or
df.withColumn("tmp",split(col("productname"),"\s+")).\
withColumn("product1",col("tmp").getItem(0)).\
withColumn("product2",col("tmp").getItem(1)).\
withColumn("product3",coalesce(col("tmp").getItem(2),lit(""))).drop("tmp").show()
#+------+-------------------+--------+--------+--------+
#| id| productname|product1|product2|product3|
#+------+-------------------+--------+--------+--------+
#|235832|EXTREME BERRY Sweet| EXTREME| BERRY| Sweet|
#| 4| BLUE CHASER SAUCE| BLUE| CHASER| SAUCE|
#| 1| LAAVA C2L5| LAAVA| C2L5| |
#+------+-------------------+--------+--------+--------+
要做更动态的方式:
df.show()
#+------+-------------------+
#| id| productname|
#+------+-------------------+
#|235832|EXTREME BERRY Sweet|
#| 4| BLUE CHASER SAUCE|
#| 1| LAAVA C2L5|
#+------+-------------------+
#caluculate array max size and store into variable
arr=int(df.select(size(split(col("productname"),"\s+")).alias("size")).orderBy(desc("size")).collect()[0][0])
#loop through arr variable and add the columns replace null with ""
(df.withColumn('temp', split('productname', '\s+')).select("*",*(coalesce(col('temp').getItem(i),lit("")).alias('product{}'.format(i+1)) for i in range(arr))).drop("temp").show())
#+------+-------------------+--------+--------+--------+
#| id| productname|product1|product2|product3|
#+------+-------------------+--------+--------+--------+
#|235832|EXTREME BERRY Sweet| EXTREME| BERRY| Sweet|
#| 4| BLUE CHASER SAUCE| BLUE| CHASER| SAUCE|
#| 1| LAAVA C2L5| LAAVA| C2L5| |
#+------+-------------------+--------+--------+--------+
关于python - 通过在空白处拆分值来创建新的 pyspark 数据框列,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/60841735/