我想在 Python 中使用 concurrent
运行一个函数。这是我的功能:
import concurrent.futures
import pandas as pd
import time
def putIndf(file):
listSel = getline(file)
datFram = savetoDataFrame(listSel)
return datFram #datatype : dataframe
def main():
newData = pd.DataFrame()
with concurrent.futures.ProcessPoolExecutor(max_workers=30) as executor:
for i,file in zip(fileList, executor.map(dp.putIndf, fileList)):
df = newData.append(file, ignore_index=True)
return df
if __name__ == '__main__':
main()
我想将数据帧连接为一个数据帧newData
,但结果只是该函数的最后一个数据帧
最佳答案
本质上,您是在每次迭代时重新分配df,并且从不增加它。您可能的意思(不明智)是初始化一个空的 df 并迭代追加:
df = pd.DataFrame()
...
df = df.append(file, ignore_index=True)
尽管如此,首选方法是构建一个数据帧集合,在循环外部一次附加在一起,并避免在循环内生成任何复杂的对象,例如数据帧。
def main():
with concurrent.futures.ProcessPoolExecutor(max_workers=30) as executor:
# LIST COMPREHENSION
df_list = [file for i,file in zip(fileList, executor.map(dp.putIndf, fileList))]
# DICTIONARY COMPREHENSION
# df_dict = {i:file for i,file in zip(fileList, executor.map(dp.putIndf, fileList))}
df = pd.concat(df_list, ignore_index=True)
return df
或者,由于池进程,将数据帧附加到列表中,在循环外仍然连接:
def main():
df_list = [] # df_dict = {}
with concurrent.futures.ProcessPoolExecutor(max_workers=30) as executor:
for i,file in zip(fileList, executor.map(dp.putIndf, fileList)):
df_list.append(file)
# df_dict[i] = file
df = pd.concat(df_list, ignore_index=True)
return df
关于python - 如何使用并发将数据帧附加到空数据帧,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/56669137/