java - 比较两个 CSV 文件并获取数据

Question

我有两个 csv 文件。一个主控 CSV 文件大约 500000 条记录。另一个 DailyCSV 文件有 50000 条记录。

DailyCSV 文件遗漏了一些必须从主 CSV 文件中获取的列。

例如

每日 CSV 文件

id,name,city,zip,occupation
1,Jhon,Florida,50069,Accountant

MasterCSV 文件

id,name,city,zip,occupation,company,exp,salary
1, Jhon, Florida, 50069, Accountant, AuditFirm, 3, $5000

我要做的是，读取两个文件，将记录与 ID 匹配，如果 ID 存在于主文件中，那么我必须获取 company, exp, salary 并将其写入新的 csv 文件。

如何实现？？

我最近做了什么

 while (true) {
            line = bstream.readLine();
            lineMaster = bstreamMaster.readLine();

            if (line == null || lineMaster == null)
            {
                break;
            }
            else
            {
                while(lineMaster != null)
                readlineSplit = line.split(",(?=([^\"]*\"[^\"]*\")*[^\"]*$)", -1);
                String splitId = readlineSplit[4];
                 String[] readLineSplitMaster =lineMaster.split(",(?=([^\"]*\"[^\"]*\")*[^\"]*$)", -1);
                  String SplitIDMaster = readLineSplitMaster[13];
                  System.out.println(splitId + "|" + SplitIDMaster);
                  //System.out.println(splitId.equalsIgnoreCase(SplitIDMaster));
                  if (splitId.equalsIgnoreCase(SplitIDMaster)) {

                      String writeLine = readlineSplit[0] + "," + readlineSplit[1] + "," + readlineSplit[2] + "," + readlineSplit[3] + "," + readlineSplit[4] + "," + readlineSplit[5] + "," + readLineSplitMaster[15]+ "," + readLineSplitMaster[16] + "," + readLineSplitMaster[17];
                      System.out.println(writeLine);
                      pstream.print(writeLine + "\r\n");
                  }
            }

        }pstream.close();
        fout.flush();
        bstream.close();
        bstreamMaster.close();

Answer 1

首先，您当前的解析方法会非常慢。使用专用于此的 CSV 解析库来加快速度。用uniVocity-parsers您可以在不到一秒的时间内处理您的 500K 记录。这是您如何使用它来解决您的问题:

首先让我们定义一些实用方法来读取/写入文件:

//opens the file for reading (using UTF-8 encoding)
private static Reader newReader(String pathToFile) {
    try {
        return new InputStreamReader(new FileInputStream(new File(pathToFile)), "UTF-8");
    } catch (Exception e) {
        throw new IllegalArgumentException("Unable to open file for reading at " + pathToFile, e);
    }
}

//creates a file for writing (using UTF-8 encoding)
private static Writer newWriter(String pathToFile) {
    try {
        return new OutputStreamWriter(new FileOutputStream(new File(pathToFile)), "UTF-8");
    } catch (Exception e) {
        throw new IllegalArgumentException("Unable to open file for writing at " + pathToFile, e);
    }
}

然后，我们可以开始读取您的每日 CSV 文件，并生成一个 map :

public static void main(String... args){
    //First we parse the daily update file.
    CsvParserSettings settings = new CsvParserSettings();
    //here we tell the parser to read the CSV headers
    settings.setHeaderExtractionEnabled(true);
    //and to select ONLY the following columns.
    //This ensures rows with a fixed size will be returned in case some records come with less or more columns than anticipated.
    settings.selectFields("id", "name", "city", "zip", "occupation");

    CsvParser parser = new CsvParser(settings);

    //Here we parse all data into a list.
    List<String[]> dailyRecords = parser.parseAll(newReader("/path/to/daily.csv"));
    //And convert them to a map. ID's are the keys.
    Map<String, String[]> mapOfDailyRecords = toMap(dailyRecords);
    ... //we'll get back here in a second.

这是从每日记录列表生成 map 的代码:

/* Converts a list of records to a map. Uses element at index 0 as the key */
private static Map<String, String[]> toMap(List<String[]> records) {
    HashMap<String, String[]> map = new HashMap<String, String[]>();
    for (String[] row : records) {
        //column 0 will always have an ID.
        map.put(row[0], row);
    }
    return map;
}

有了记录图，我们可以处理您的主文件并生成更新列表:

private static List<Object[]> processMasterFile(final Map<String, String[]> mapOfDailyRecords) {
    //we'll put the updated data here
    final List<Object[]> output = new ArrayList<Object[]>();

    //configures the parser to process only the columns you are interested in.
    CsvParserSettings settings = new CsvParserSettings();
    settings.setHeaderExtractionEnabled(true);
    settings.selectFields("id", "company", "exp", "salary");

    //All parsed rows will be submitted to the following RowProcessor. This way the bigger Master file won't
    //have all its rows stored in memory.
    settings.setRowProcessor(new AbstractRowProcessor() {
        @Override
        public void rowProcessed(String[] row, ParsingContext context) {
            // Incoming rows from MASTER will have the ID as index 0.
            // If the daily update map contains the ID, we'll get the daily row
            String[] dailyData = mapOfDailyRecords.get(row[0]);
            if (dailyData != null) {
                //We got a match. Let's join the data from the daily row with the master row.
                Object[] mergedRow = new Object[8];

                for (int i = 0; i < dailyData.length; i++) {
                    mergedRow[i] = dailyData[i];
                }
                for (int i = 1; i < row.length; i++) { //starts from 1 to skip the ID at index 0
                    mergedRow[i + dailyData.length - 1] = row[i];
                }
                output.add(mergedRow);
            }
        }
    });

    CsvParser parser = new CsvParser(settings);
    //the parse() method will submit all rows to the RowProcessor defined above.
    parser.parse(newReader("/path/to/master.csv"));

    return output;
}

最后，我们可以获得合并后的数据并将所有内容写入另一个文件:

    ... // getting back to the main method here
    //Now we process the master data and get a list of updates
    List<Object[]> updatedData = processMasterFile(mapOfDailyRecords);

    //And write the updated data to another file
    CsvWriterSettings writerSettings = new CsvWriterSettings();
    writerSettings.setHeaders("id", "name", "city", "zip", "occupation", "company", "exp", "salary");
    writerSettings.setHeaderWritingEnabled(true);

    CsvWriter writer = new CsvWriter(newWriter("/path/to/updates.csv"), writerSettings);
    //Here we write everything, and get the job done.
    writer.writeRowsAndClose(updatedData);
}

这应该很有魅力。希望对您有所帮助。

披露:我是这个图书馆的作者。它是开源且免费的(Apache V2.0 许可)。