python - 在mysql加载数据文件导入期间删除CSV公式值

Question

因此，我尝试使用 load data local infile 命令将 .csv 文件直接导入 mysql，但我遇到了 excel 公式字段残留的字段，我不知道如何操作摆脱他们。下面示例中第一个字段的内容前面有一个 =。

表结构设置为允许第一个字段为 VARCHAR(100)，但如果可能的话，我希望将其设为 INT。以下是正在上传的 csv 内容的示例。

"MID","DBA Name","Partner ID","Partner Name","Sub Partner ID","Sub Partner Name","Active Months","Bonus Amount","Bonus Applied Date","Partner Percentage","Partner Share","Total Payment"
="0008788014065741","company2","7968","me,"11839","Joe Blow","0","$50.00","","","","$350.64"
="0008788014065756","company2","7968","you","11839","Joe Blow","0","$50.00","","","","$294.60"

这是我用来导入数据的 mysql load 命令:

sql = """
    LOAD DATA LOCAL INFILE '%(upload)s' IGNORE INTO TABLE `%(table)s`
    FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' 
    LINES TERMINATED BY '\\r\\n' 
    IGNORE 1 LINES ;
      """ % {"upload": file, "table": report}
self.db.query( sql )

是否可以通过正则表达式或其他方式对导入执行某些操作？我不知道，我只是捕获救命稻草......

感谢您的投入!

Answer 1

您可以使用LOAD DATA INFILE通过两种方式完成此操作。

First 按原样读取第一个字段值，然后在 SET 子句中去掉等号 = 和双引号。此外，您很可能希望在加载数据时进行其他转换，例如:

• 当字段为空时设置实际的NULL
• 从货币值(value)中去除美元符号
• 您可能必须转换日期值(但您的示例数据没有它们，因此没有可以推断的信息)

LOAD DATA LOCAL INFILE '/path/to/your/file.csv' 
IGNORE INTO TABLE table_name
    FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' 
    LINES TERMINATED BY '\r\n'
    IGNORE 1 LINES 
(@MID, DBAName, PartnerID, PartnerName, SubPartnerID, SubPartnerName, ActiveMonths,
 @BonusAmount, @BonusAppliedDate, @PartnerPercentage, @PartnerShare, @TotalPayment)
SET MID = TRIM(BOTH '"' FROM SUBSTR(@MID, 2)), -- here we get rid of equal sign and double quotes
    BonusAmount  = TRIM(LEADING '$' FROM NULLIF(@BonusAmount, '')),
    BonusAppliedDate = NULLIF(@BonusAppliedDate, ''),
    PartnerPercentage = NULLIF(@PartnerPercentage, ''),
    PartnerShare = TRIM(LEADING '$' FROM NULLIF(@PartnerShare, '')),
    TotalPayment = TRIM(LEADING '$' FROM NULLIF(@TotalPayment, ''))

第二种方法是利用LINES STARTING BY子句

LOAD DATA LOCAL INFILE '/path/to/your/file.csv' 
IGNORE INTO TABLE table_name
    FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' 
    LINES TERMINATED BY '\r\n' STARTING BY '='
    IGNORE 1 LINES 
(MID, DBAName, PartnerID, PartnerName, SubPartnerID, SubPartnerName, ActiveMonths, @BonusAmount, BonusAppliedDate, PartnerPercentage, PartnerShare, @TotalPayment)
SET BonusAmount  = TRIM(LEADING '$' FROM NULLIF(@BonusAmount, '')),
    BonusAppliedDate = NULLIF(@BonusAppliedDate, ''),
    PartnerPercentage = NULLIF(@PartnerPercentage, ''),
    PartnerShare = TRIM(LEADING '$' FROM NULLIF(@PartnerShare, '')),
    TotalPayment = TRIM(LEADING '$' FROM NULLIF(@TotalPayment, ''))

<小时/>

现在，如果您的目标表架构看起来像这样

CREATE TABLE table_name 
(
    MID BIGINT, 
    DBAName           VARCHAR(100),
    PartnerID         INT,
    PartnerName       VARCHAR(100),
    SubPartnerID      INT,
    SubPartnerName    VARCHAR(100),
    ActiveMonths      INT,
    BonusAmount       DECIMAL(19, 2),
    BonusAppliedDate  DATE,
    PartnerPercentage DECIMAL(3, 2),
    PartnerShare      DECIMAL(19, 2),
    TotalPayment      DECIMAL(19, 2)
);

然后使用任一方法加载后，我们在表格中得到的内容

mysql> select * from table_name;
+---------------+----------+-----------+-------------+--------------+----------------+--------------+-------------+------------------+-------------------+--------------+--------------+
| MID           | DBAName  | PartnerID | PartnerName | SubPartnerID | SubPartnerName | ActiveMonths | BonusAmount | BonusAppliedDate | PartnerPercentage | PartnerShare | TotalPayment |
+---------------+----------+-----------+-------------+--------------+----------------+--------------+-------------+------------------+-------------------+--------------+--------------+
| 8788014065741 | company2 |      7968 | me          |        11839 | Joe Blow       |            0 |       50.00 | NULL             |              NULL |         NULL |       350.64 |
| 8788014065756 | company2 |      7968 | you         |        11839 | Joe Blow       |            0 |       50.00 | NULL             |              NULL |         NULL |       294.60 |
+---------------+----------+-----------+-------------+--------------+----------------+--------------+-------------+------------------+-------------------+--------------+--------------+
2 rows in set (0.00 sec)