我有不包含标题的 CSV 文件,每行仅包含 2 列(时间和数据名称),并且始终包含数据,其余行的列数根据数据而变化。
我已成功将“正常”CSV 文件导入到具有一致列数的 pandas 中。它工作得非常好,但我在文档中看到了处理当前情况的任何内容。
以下是相关 CSV 文件的片段:
1573081480.942000, /eeg, 843.3333, 854.61536, 851.79486, 849.3773, 863.0769
1573081480.942000, /eeg, 844.1392, 857.4359, 849.3773, 861.8681, 890.07324
1573081480.943000, /eeg, 853.8095, 853.8095, 850.989, 866.30035, 854.61536
1573081480.944000, /eeg, 855.42126, 855.0183, 846.1539, 852.1978, 846.9597
1573081480.947000, /eeg, 844.1392, 853.8095, 846.55676, 842.52747, 873.5531
1573081480.947000, /eeg, 848.97437, 853.00366, 851.79486, 853.00366, 856.2271
1573081480.948000, /eeg, 859.0476, 852.6007, 850.18317, 863.8828, 826.0073
1573081480.950000, /eeg, 859.0476, 851.79486, 853.00366, 866.30035, 819.5604
1573081480.950000, /eeg, 851.79486, 852.1978, 846.9597, 854.61536, 859.45056
1573081480.951000, /eeg, 856.63007, 853.00366, 846.55676, 840.9158, 854.21246
1573081480.960000, /elements/alpha_absolute, 0.48463312
1573081480.960000, /elements/beta_absolute, 0.061746284
1573081480.961000, /elements/gamma_absolute, 0.7263172
1573081480.961000, /elements/theta_absolute, 0.7263172
1573081480.961000, /elements/delta_absolute, 0.7263172
我需要的结果看起来像这样
time, eeg_0, eeg_1, eeg_2, eeg_3, delta, theta, alpha, beta, gamma
1573081480.942000, 844.1392, 857.4359, 849.3773, 861.8681,,,,,
1573081480.947000, 844.1392, 853.8095, 846.55676, 842.52747, 873.5531,,,,,
1573081480.947000, 848.97437, 853.00366, 851.79486, 853.00366, 856.2271,,,,,
1573081480.948000, 859.0476, 852.6007, 850.18317, 863.8828, 826.0073,,,,,
1573081480.960000,,,,,,,0.48463312,,
1573081480.960000,,,,,,,,0.061746284,
1573081480.961000,,,,,0.7263172,,,,
1573081480.961000,,,,,0.52961296,,,
1573081480.962000,,,,,,,,-0.26484978
如您所见,值的数量可能会根据存储的数据而变化。
我希望导入过程与“普通”CSV 文件一样简单且高效。
这是我希望避免的,它非常冗长且低效:
d = {
'time': [0.],
'eeg0': [0.],'eeg1': [0.],'eeg2': [0.],'eeg3': [0.],'eeg4': [0.],
'delta_absolute': [0.], 'theta_absolute': [0], 'alpha_absolute': [0], 'beta_absolute': [0], 'alpha_absolute': [0],
'acc0': [0], 'acc1': [0], 'acc2': [0], 'gyro0': [0], 'gyro1': [0], 'gyro2': [0],
'concentration': [0],'mellow': [0]
}
df_new_data = pd.DataFrame(data=d)
csvfile = open(fname)
csv_reader = csv.reader(csvfile, delimiter=',')
csv_data = list(csv_reader)
row_count = len(csv_data)
for row in csv_data:
if row[1] == ' /muse/acc':
df_new_data = df_new_data.append({'acc0' : row[2], 'acc1' : row[3], 'acc2' : row[4]}, ignore_index=True)
if row[1] == ' /muse/gyro':
df_new_data = df_new_data.append({'gyro0' : row[2], 'gyro1' : row[3], 'gyro2' : row[4]}, ignore_index=True)
编辑:
我发现,如果 CSV 文件的第一行包含的字段较少,则任何后续行都将失败。上面的 CSV 数据示例有效,但这个示例无效:
573081480.960000, /elements/alpha_absolute, 0.48463312
1573081480.960000, /elements/beta_absolute, 0.061746284
1573081480.961000, /elements/gamma_absolute, 0.7263172
1573081480.961000, /elements/theta_absolute, 0.7263172
1573081480.961000, /elements/delta_absolute, 0.7263172
1573081480.942000, /eeg, 843.3333, 854.61536, 851.79486, 849.3773, 863.0769
1573081480.942000, /eeg, 844.1392, 857.4359, 849.3773, 861.8681, 890.07324
1573081480.943000, /eeg, 853.8095, 853.8095, 850.989, 866.30035, 854.61536
1573081480.944000, /eeg, 855.42126, 855.0183, 846.1539, 852.1978, 846.9597
1573081480.947000, /eeg, 844.1392, 853.8095, 846.55676, 842.52747, 873.5531
1573081480.947000, /eeg, 848.97437, 853.00366, 851.79486, 853.00366, 856.2271
1573081480.948000, /eeg, 859.0476, 852.6007, 850.18317, 863.8828, 826.0073
1573081480.950000, /eeg, 859.0476, 851.79486, 853.00366, 866.30035, 819.5604
1573081480.950000, /eeg, 851.79486, 852.1978, 846.9597, 854.61536, 859.45056
1573081480.951000, /eeg, 856.63007, 853.00366, 846.55676, 840.9158, 854.21246
pandas 会产生这个错误:
pandas.errors.ParserError: Error tokenizing data. C error: Expected 3 fields in line 6, saw 7
提前致谢!
最佳答案
目前尚不清楚您到底想要什么。很高兴您提供了示例输出,但如果它是您输入的实际预期输出,那就容易多了。
据我了解,最简单的方法是循环每种类型,查找它们使用了多少列,创建许多框架,最后连接它们。像这样:
# Using pandas:
max_number_of_columns = pandas.read_csv('test.txt', sep='|', header=None)[0].str.count(',').max()
# or just hardcoded:
max_number_of_columns = 10
base = pandas.read_csv('test.txt', header=None, names=list(range(max_number_of_columns)))
base.columns = ['time','datatype'] + list(base.columns[2:])
results = [base.iloc[:,:2]]
for datatype in base['datatype'].unique():
group = base[base['datatype']==datatype].iloc[:,2:].dropna(how='all', axis=1)
group.columns = [f"{datatype}_{x}" for x in range(len(group.columns))]
results.append(group)
final = pandas.concat(results, axis=1)
编辑:修复第一行包含的列少于后面的行的情况。
关于python - 将 CSV 文件转换为 pandas 的 'flat file",我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/58786303/