multiprocessing - 多处理命令列表

Question

我正在尝试运行命令列表。问题是该列表可能会很长，因此同时运行多个命令会很棒。

如何使用多处理模块执行此操作？

list_of_commands = [cmd foo, cmd bar, ...]

main_log_file = open( os.getcwd() + '/Error.log', 'w+')

Count = 0
for Job in list_of_commands:
    Count += 1
    child = subprocess.Popen(Job, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    streamdata = child.communicate()[0]
    errcode = child.returncode
    if errcode == 0:
        print ( 'Job', Count, 'Success' )
    elif errcode == 1:
        print ( 'Job', Count, 'Completed With Errors' )
    elif errcode == 2:
        print ( 'Job', Count, 'Error' )
        main_log_file.write ( streamdata.decode('ascii') + str(errcode) + '\n' )

main_log_file.close()

执行顺序并不重要

Answer 1

您可以使用concurrent.futures.ThreadPoolExecutor map 函数来运行恒定数量的并行执行。

WORKERS = 5  # amount of concurrent executions you want

def launcher(job):
    child = subprocess.Popen(job, ... )
    streamdata = child.communicate()[0]
    ...

with concurrent.futures.ThreadPoolExecutor(max_workers=WORKERS) as pool:
    pool.map(launcher, jobs)