Python 多处理 : kill process if it is taking too long to return

标签 python parallel-processing multiprocessing

下面是一个简单的例子,因为一个子进程没有返回任何东西就退出了,父进程一直在等待,所以卡住了。如果花费的时间太长,是否有办法使进程超时并让其余的继续?我是 python 多处理的初学者,我发现文档不是很有启发性。

import multiprocessing as mp
import time

def foo(x):
    if x == 3:
        sys.exit()
    #some heavy computation here
    return result

if __name__ == '__main__':  
    pool = mp.Pool(mp.cpu_count)
    results = pool.map(foo, [1, 2, 3])

最佳答案

我遇到了同样的问题,我就是这样解决的。也许有更好的解决方案,但是,它也解决了未提及的问题。例如。如果该进程占用大量资源,则正常终止可能需要一段时间才能完成该进程——因此我使用强制终止 (kill -9)。这部分可能仅适用于 Linux,因此如果您使用其他操作系统,您可能需要调整终止。

它是我自己代码的一部分,因此可能无法复制粘贴。

from multiprocessing import Process, Queue
import os 
import time 

timeout_s = 5000 # seconds after which you want to kill the process

queue = Queue()  # results can be written in here, if you have return objects

p = Process(target=INTENSIVE_FUNCTION, args=(ARGS_TO_INTENSIVE_FUNCTION, queue))
p.start()

start_time = time.time()
check_interval_s = 5  # regularly check what the process is doing

kill_process = False
finished_work = False

while not kill_process and not finished_work:
    time.sleep(check_interval_s)  
    now = time.time()
    runtime = now - start_time

    if not p.is_alive():
        print("finished work")
        finished_work = True

    if runtime > timeout_s and not finished_work:
        print("prepare killing process")
        kill_process = True

if kill_process:
    while p.is_alive():
        # forcefully kill the process, because often (during heavvy computations) a graceful termination
        # can be ignored by a process.
        print(f"send SIGKILL signal to process because exceeding {timeout_s} seconds.")
        os.system(f"kill -9 {p.pid}")

        if p.is_alive():
            time.sleep(check_interval_s)
else:
    try:
        p.join(60)  # wait 60 seconds to join the process
        RETURN_VALS = queue.get(timeout=60)
    except Exception:
        # This can happen if a process was killed for other reasons (such as out of memory)
        print("Joining the process and receiving results failed, results are set as invalid.")

关于Python 多处理 : kill process if it is taking too long to return,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/59808975/

相关文章:

python - 使用pdb调试python脚本时有没有获取详细对象信息的方法?

python - Altair - 在图中画一条线,其中 x = y

java - 如何在 Java 中以最高效和优雅的方式使用并行处理

python - 多处理奇怪的行为?

Python MongoDB (PyMongo) 多重处理游标

python - 为什么使用 SqlAlchemy func.lower(string) 与 string.lower()?

python - Django 的后台作业/任务监视器

r - parLapply 在 R6 类别内

r - 为什么foreach%dopar%随每个其他节点变慢?

linux - 如果进程数大于内核数的一半,为什么性能会下降?