如何在 Python 中读取和更新多个 worker 之间共享的变量？
例如，我在 Python 中使用多个进程扫描文件列表，并想检查父目录是否已被扫描。
def readFile(filename):
""" Add the parent folder to the database and process the file
"""
path_parts = os.path.split(filename)
dirname = os.path.basename(path_parts[0])
if dirname not in shared_variable:
# Insert into the database
#Other file functions
def main():
""" Walk through files and pass each file to readFile()
"""
queue = multiprocessing.Queue()
pool = multiprocessing.Pool(None, init, [queue])
for dirpath, dirnames, filenames in os.walk(PATH):
full_path_fnames = map(lambda fn: os.path.join(dirpath, fn),
filenames)
pool.map(readFile, full_path_fnames)
您可以使用 multiprocessing.Manager 来帮助解决这个问题。它允许您创建一个可以在进程之间共享的列表:
from functools import partial
import multiprocessing
def readFile(shared_variable, filename):
""" Add the parent folder to the database and process the file
"""
path_parts = os.path.split(filename)
dirname = os.path.basename(path_parts[0])
if dirname not in shared_variable:
# Insert into the database
#Other file functions
def main():
""" Walk through files and pass each file to readFile()
"""
manager = multiprocessing.Manager()
shared_variable = manager.list()
queue = multiprocessing.Queue()
pool = multiprocessing.Pool(None, init, [queue])
func = partial(readFile, shared_variable)
for dirpath, dirnames, filenames in os.walk(PATH):
full_path_fnames = map(lambda fn: os.path.join(dirpath, fn),
filenames)
pool.map(func, full_path_fnames)
partial 只是用来更容易地将 shared_variable 与 的每个成员一起传递给 readFile 的每个调用full_path_fnames 通过 map。