我有一系列非常困惑的 *.csv 文件正在由 pandas 读取。一个 csv 示例是:
Instrument 35392
"Log File Name : station"
"Setup Date (MMDDYY) : 031114"
"Setup Time (HHMMSS) : 073648"
"Starting Date (MMDDYY) : 031114"
"Starting Time (HHMMSS) : 090000"
"Stopping Date (MMDDYY) : 031115"
"Stopping Time (HHMMSS) : 235959"
"Interval (HHMMSS) : 010000"
"Sensor warmup (HHMMSS) : 000200"
"Circltr warmup (HHMMSS) : 000200"
"Date","Time","","Temp","","SpCond","","Sal","","IBatt",""
"MMDDYY","HHMMSS","","øC","","mS/cm","","ppt","","Volts",""
"Random message here 031114 073721 to 031114 083200"
03/11/14,09:00:00,"",15.85,"",1.408,"",.74,"",6.2,""
03/11/14,10:00:00,"",15.99,"",1.96,"",1.05,"",6.3,""
03/11/14,11:00:00,"",14.2,"",40.8,"",26.12,"",6.2,""
03/11/14,12:00:01,"",14.2,"",41.7,"",26.77,"",6.2,""
03/11/14,13:00:00,"",14.5,"",41.3,"",26.52,"",6.2,""
03/11/14,14:00:00,"",14.96,"",41,"",26.29,"",6.2,""
"message 3"
"message 4"**
我一直在使用此代码导入 *csv 文件,处理双标题,拉出空列,然后删除包含错误数据的违规行:
DF = pd.read_csv(BADFILE,parse_dates={'Datetime_(ascii)': [0,1]}, sep=",", \
header=[10,11],na_values=['','na', 'nan nan'], \
skiprows=[10], encoding='cp1252')
DF = DF.dropna(how="all", axis=1)
DF = DF.dropna(thresh=2)
droplist = ['message', 'Random']
DF = DF[~DF['Datetime_(ascii)'].str.contains('|'.join(droplist))]
DF.head()
Datetime_(ascii) (Temp, øC) (SpCond, mS/cm) (Sal, ppt) (IBatt, Volts)
0 03/11/14 09:00:00 15.85 1.408 0.74 6.2
1 03/11/14 10:00:00 15.99 1.960 1.05 6.3
2 03/11/14 11:00:00 14.20 40.800 26.12 6.2
3 03/11/14 12:00:01 14.20 41.700 26.77 6.2
4 03/11/14 13:00:00 14.50 41.300 26.52 6.2
这工作得很好,直到我有一个文件在标题后有一个错误的 1 行行:“这里随机消息 031114 073721 到 031114 083200”
我收到的错误是:
*C:\Users\USER\AppData\Local\Continuum\Anaconda3\lib\site-
packages\pandas\io\parsers.py in _do_date_conversions(self, names, data)
1554 data, names = _process_date_conversion(
1555 data, self._date_conv, self.parse_dates, self.index_col,
-> 1556 self.index_names, names,
keep_date_col=self.keep_date_col)
1557
1558 return names, data
C:\Users\USER\AppData\Local\Continuum\Anaconda3\lib\site-
packages\pandas\io\parsers.py in _process_date_conversion(data_dict,
converter, parse_spec, index_col, index_names, columns, keep_date_col)
2975 if not keep_date_col:
2976 for c in list(date_cols):
-> 2977 data_dict.pop(c)
2978 new_cols.remove(c)
2979
KeyError: ('Time', 'HHMMSS')*
如果我删除该行,代码就可以正常工作。同样,如果我删除 header= 行,代码就可以正常工作。但是,我希望能够保留它,因为我正在阅读数百个这样的文件。
困难:我不想在调用pandas.read_csv()之前打开每个文件,因为这些文件可能相当大 - 因此我不想多次读取和保存!另外,我更喜欢真正的 pandas/pythonic 解决方案,该解决方案不涉及首先将文件作为 stringIO 缓冲区打开以删除有问题的行。
最佳答案
这是一种方法,利用 skip_rows
接受可调用函数这一事实。该函数仅接收正在考虑的行索引,这是该参数的内置限制。
因此,可调用函数skip_test()
首先检查当前索引是否在要跳过的已知索引集中。如果不匹配,则它打开实际文件并检查相应的行以查看其内容是否匹配。
skip_test()
函数有点 hacky,因为它确实检查实际文件,尽管它只检查直到它正在评估的当前行索引。它还假设坏行始终以相同的字符串开头(在示例中为 "foo"
),但这似乎是给定 OP 的安全假设。
# example data
""" foo.csv
uid,a,b,c
0,1,2,3
skip me
1,11,22,33
foo
2,111,222,333
"""
import pandas as pd
def skip_test(r, fn, fail_on, known):
if r in known: # we know we always want to skip these
return True
# check if row index matches problem line in file
# for efficiency, quit after we pass row index in file
f = open(fn, "r")
data = f.read()
for i, line in enumerate(data.splitlines()):
if (i == r) & line.startswith(fail_on):
return True
elif i > r:
break
return False
fname = "foo.csv"
fail_str = "foo"
known_skip = [2]
pd.read_csv(fname, sep=",", header=0,
skiprows=lambda x: skip_test(x, fname, fail_str, known_skip))
# output
uid a b c
0 0 1 2 3
1 1 11 22 33
2 2 111 222 333
如果您确切地知道随机消息出现时将出现在哪一行,那么这会快得多,因为您可以告诉它不要检查文件内容以查找超过潜在违规行的任何索引。
关于python - 忽略 pandas.read_csv() 中破坏 header= keywords 的坏数据行,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/45679857/