python - 在 Python 中比较黄金标准 csv 文件和提取值 csv 文件

Question

这是一项数据挖掘任务，我们将自动对提取的质量进行评分。 有一个黄金标准 csv，可能包含如下所示的字段

golden_standard.csv

| id | description             | amount  | date       |
|----|-------------------------|---------|------------|
| 1  | Some description.       | $150.54 | 12/12/2012 |
| 2  | Some other description. | $200    | 10/10/2015 |
| 3  | Other description.      | $25     | 11/11/2014 |
| 4  | My description          | $11.35  | 01/01/2015 |
| 5  | Your description.       | $20     | 03/03/2013 |

，然后有两个可能的提取结果文件:

提取1.csv

| id | description             | date       |
|----|-------------------------|------------|
| 1  | Some description.       | 12/12/2012 |
| 2  | Some other description. | 10/10/2015 |
| 3  | Other description.      | 11/11/2014 |
| 4  | 122333222233332221      | 11/11/2014 |
| 5  | Your description.       | 03/03/2013 |

提取2.csv

| id | description             | amount  | date       |
|----|-------------------------|---------|------------|
| 1  | Some description.       | $150.54 | 12/12/2012 |
| 2  | Some other description. | $200    | 10/10/2015 |
| -  | ----------------------- | -----   | ---------- |
| 5  | Your description.       | $20     | 03/03/2013 |

extract3.csv

| Garbage  | More Garbage       |
| Garbage  | More Garbage       | 

我希望我的程序报告提取 1 缺少一列，并且第 2 列中的值未正确匹配。

对于第二种情况，我缺少条目并且某些行都不匹配。

在最后一种情况下，生成的 csv 完全搞砸了，但我仍然希望程序能够检测到一些有意义的异常。

有人有一些快速而聪明的方法在Python中进行这种比较吗？

我有我可以在此处发布的常规、较长的逐行和逐列迭代方式，但我认为可能有一种更快、更优雅的 Python 方式来做到这一点。

非常感谢任何帮助。

Answer 1

免责声明:我的方法使用 pandas 库。

首先，数据设置。

gold_std.csv

id,description,amount,date
1,Some description.,$150.54,12/12/2012
2,Some other description.,$200,10/10/2015
3,Other description.,$25,11/11/2014
4,My description,$11.35,01/01/2015
5,Your description.,$20,03/03/2013

extract1.csv

id,description,date
1,Some description.,12/12/2012
2,Some other description.,10/10/2015
3,Other description.,11/11/2014
4,122333222233332221,11/11/2014
5,Your description.,03/03/2013

extract2.csv

id,description,amount,date
1,Some description.,$150.54,12/12/2012
2,Some other description.,$200,10/10/2015
3,Other description.,$25,11/11/2014
5,Your description.,$20,03/03/2013

第二，代码。

import pandas as pd

def compare_extract(extract_name, reference='gold_std.csv'):

    gold = pd.read_csv(reference)
    ext = pd.read_csv(extract_name)

    gc = set(gold.columns)
    header = ext.columns
    extc = set(header)

    if gc != extc:
        missing = ", ".join(list(gc - extc))
        print "Extract has the following missing columns: {}".format(missing)
    else:
        print "Extract has the same column as standard. Checking for abberant rows..."
        gold_list = gold.values.tolist()
        ext_list = ext.values.tolist()
        # Somewhat non-pandaic approach because possible no same IDs so we're relying
        # on set operations instead. A bit hackish, actually.
        diff = list(set(map(tuple, gold_list)) - set(map(tuple, ext_list)))
        df = pd.DataFrame(diff, columns=header)
        print "The following rows are not in the extract: "
        print df

第三，测试运行。

e1 = 'extract1.csv'
compare_extract(e1)
# Extract has the following missing columns: amount

e2 = 'extract2.csv'
compare_extract(e2)
# Extract has the same column as standard. Checking for abberant rows...
# The following rows are not in the extract: 
#    id     description  amount        date
# 0   4  My description  $11.35  01/01/2015

最后，最后的摘录有点随意。我认为对于这个问题，你最好编写一个非 pandas 算法。