python - 在 Python 中并行运行多个外部命令

Question

我想通过 Python 同时运行多个外部命令(例如打开 notepad.exe)。我知道如何使用 subprocess 模块来做到这一点。但是，我还想同时并行运行多个此类命令，并确保在任何时候，n 个命令都用完总共 m 个命令，直到所有 m 个命令都运行完毕。命令运行完毕。下面是示例代码/解释来说明这一点，其中 n = 3 和 m = 10

import subprocess

commands = []
batch_size = 3

for i in range(10):
    commands.append('notepad.exe')

def run_in_batches(batch_size):

    # run multiple( = batch_size = 3) commands in parallel through subprocess.Popen
    # command1 = subprocess.Popen(commands[0])
    # command2 = subprocess.Popen(commands[1])
    # command3 = subprocess.Popen(commands[2])
    # command1.wait()
    # command2.wait()
    # command3.wait()
    # if command1.poll() == 0 or commad2.poll() == 0 or command3.poll == 0:
    #   queue the next command in commands
    #   perform this check repeatedly so that 3 commands are running at any time untill all commands are finished running

如您所见，我一直在尝试使用 subprocess.Popen 来执行此操作，但我无法正确编码。我真的很挣扎，其中我必须确保 3 个命令始终运行，如果其中任何一个命令完成，只有然后执行队列中的下一个命令，直到所有命令都成功运行。任何帮助将不胜感激，谢谢!

Answer 1

你应该使用Threadpool为此，使用 subprocess.run因为它会阻塞正在运行的线程。

import subprocess
from multiprocessing.pool import ThreadPool
commands = []
batch_size = 3

for i in range(10):
    commands.append('notepad.exe')

def run_in_batches(batch_size, commands_to_run):
    with ThreadPool(batch_size) as pool:
        pool.map(subprocess.run, commands_to_run)

run_in_batches(batch_size, commands)