amazon-web-services - 在 s3 中重命名 Pyspark 输出文件

Question

我使用以下命令将 pyspark 数据帧保存到 s3:

df.coalesce(1).write.partitionBy('DATE'
                                    ).format("com.databricks.spark.csv"
                                    ).mode('overwrite'
                                    ).option("header", "true"
                                    ).save(output_path)

哪个给我:

file_path/FLORIDA/DATE=2019-04-29/part-00000-1691d1c6-2c49-4cbe-b454-d0165a0d7bde.c000.csv
file_path/FLORIDA/DATE=2019-04-30/part-00000-1691d1c6-2c49-4cbe-b454-d0165a0d7bde.c000.csv
file_path/FLORIDA/DATE=2019-05-01/part-00000-1691d1c6-2c49-4cbe-b454-d0165a0d7bde.c000.csv
file_path/FLORIDA/DATE=2019-05-02/part-00000-1691d1c6-2c49-4cbe-b454-d0165a0d7bde.c000.csv

是否有一种简单的方法可以在 s3 中重新格式化此路径以遵循此结构？:

file_path/FLORIDA/allocation_FLORIDA_20190429.csv
file_path/FLORIDA/allocation_FLORIDA_20190430.csv
file_path/FLORIDA/allocation_FLORIDA_20190501.csv
file_path/FLORIDA/allocation_FLORIDA_20190502.csv

我有几千个这样的东西，所以如果有一种编程方式可以做到这一点，那将不胜感激!

Answer 1

想出了一个不错的方法来解决这个问题:

import datetime
import boto3
s3 = boto3.resource('s3')

for i in range(5):
    date = datetime.datetime(2019,4,29)
    date += datetime.timedelta(days=i)
    date = date.strftime("%Y-%m-%d")
    print(date)
    old_date = 'file_path/FLORIDA/DATE={}/part-00000-1691d1c6-2c49-4cbe-b454-d0165a0d7bde.c000.csv'.format(date)
    print(old_date)
    date = date.replace('-','')
    new_date = 'file_path/FLORIDA/allocation_FLORIDA_{}.csv'.format(date)
    print(new_date)

    s3.Object('my_bucket', new_date).copy_from(CopySource='my_bucket/' + old_date)

    s3.Object('my_bucket', old_date).delete()