我正在运行一个循环遍历嵌套字典的基本脚本,从每条记录中获取数据,并将其附加到 Pandas DataFrame。数据看起来像这样:
data = {"SomeCity": {"Date1": {record1, record2, record3, ...}, "Date2": {}, ...}, ...}
它总共有几百万条记录。脚本本身如下所示:
city = ["SomeCity"]
df = DataFrame({}, columns=['Date', 'HouseID', 'Price'])
for city in cities:
for dateRun in data[city]:
for record in data[city][dateRun]:
recSeries = Series([record['Timestamp'],
record['Id'],
record['Price']],
index = ['Date', 'HouseID', 'Price'])
FredDF = FredDF.append(recSeries, ignore_index=True)
然而,这运行起来非常慢。在我寻找一种并行化方法之前,我只是想确保我没有遗漏一些明显的东西,这些东西会使它按原样执行得更快,因为我对 Pandas 还是很陌生。
最佳答案
我还在一个循环中使用了数据帧的append函数,我很困惑它运行得有多慢。
基于本页上的正确答案,对于那些正在受苦的人来说是一个有用的例子。
Python 版本:3
Pandas 版本:0.20.3
# the dictionary to pass to pandas dataframe
d = {}
# a counter to use to add entries to "dict"
i = 0
# Example data to loop and append to a dataframe
data = [{"foo": "foo_val_1", "bar": "bar_val_1"},
{"foo": "foo_val_2", "bar": "bar_val_2"}]
# the loop
for entry in data:
# add a dictionary entry to the final dictionary
d[i] = {"col_1_title": entry['foo'], "col_2_title": entry['bar']}
# increment the counter
i = i + 1
# create the dataframe using 'from_dict'
# important to set the 'orient' parameter to "index" to make the keys as rows
df = DataFrame.from_dict(d, "index")
“from_dict”函数:https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.from_dict.html
关于python - 提高 Pandas DataFrames 的行追加性能,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/27929472/