java - 将包含百万值的 csv 文件上传到 mysql 中的特定列

Question

数据库:MySQL

CSV 文件有大约百万条记录，如下所示:

234546,345674,464368,563236,684269,707853,...

数据应按以下方式添加到数据库中:

id|number|status| ...
-----------------------
 1|234546| ...  | ...
 2|345674| ...  | ...
 3|464368| ...  | ...
 4|563236| ...  | ...
 5|684269| ...  | ...
 6|707853| ...  | ...

当 csv 文件包含每一列的值时，我发现了一些上传 csv 文件的方法。但就我而言，我必须将值插入特定列。在 Java 中执行此操作的有效方法是什么？

Answer 1

从文件加载数据最快的方法是 LOAD DATA INFILE 。试试这个方法

LOAD DATA LOCAL INFILE '/path/to/your/file.csv'
IGNORE -- use this if you have duplicate numbers in the file and UNIQUE constraint defined on `number` column
INTO TABLE table1
FIELDS TERMINATED BY ','
LINES TERMINATED BY ','
(number)

让我们试试吧

mysql> CREATE TABLE Table1
    -> (
    ->   `id` int not null auto_increment primary key, 
    ->   `number` int, 
    ->   `status` varchar(32)
    -> );
Query OK, 0 rows affected (0.13 sec)

mysql> 
mysql> LOAD DATA LOCAL INFILE '/tmp/test.csv'
    -> INTO TABLE table1
    -> FIELDS TERMINATED BY ','
    -> LINES TERMINATED BY ','
    -> (number);
Query OK, 6 rows affected (0.00 sec)
Records: 6  Deleted: 0  Skipped: 0  Warnings: 0

mysql> select * from table1;
+----+--------+--------+
| id | number | status |
+----+--------+--------+
|  1 | 234546 | NULL   |
|  2 | 345674 | NULL   |
|  3 | 464368 | NULL   |
|  4 | 563236 | NULL   |
|  5 | 684269 | NULL   |
|  6 | 707853 | NULL   |
+----+--------+--------+
6 rows in set (0.00 sec)

---

UPDATE: If you need to do some basic transformations or validations you can use SET clause along with user variables

LOAD DATA LOCAL INFILE '/path/to/your/file.csv'
IGNORE INTO TABLE table1
FIELDS TERMINATED BY ','
LINES TERMINATED BY ','
(@number)
SET number = CONCAT('prefix', @number)