python - 多处理池意外杀死线程

Question

我在下面的 python 3.6.3 中编写并执行了一个简单的测试程序。它正在具有 4 个内核的机器上执行。

import multiprocessing
import time

def f(num):
  print(multiprocessing.current_process(), num)
  time.sleep(1)
  if (num % 2):
    raise Exception


pool = multiprocessing.Pool(5)

try:
  pool.map(f, range(1,20))
except Exception as e:
  print("EXCEPTION")

pool.close()
pool.join()

pool = multiprocessing.Pool(5) 的输出:

<ForkProcess(ForkPoolWorker-1, started daemon)> 1
<ForkProcess(ForkPoolWorker-2, started daemon)> 2
<ForkProcess(ForkPoolWorker-3, started daemon)> 3
<ForkProcess(ForkPoolWorker-4, started daemon)> 4
<ForkProcess(ForkPoolWorker-5, started daemon)> 5
<ForkProcess(ForkPoolWorker-2, started daemon)> 6
<ForkProcess(ForkPoolWorker-1, started daemon)> 7
<ForkProcess(ForkPoolWorker-4, started daemon)> 8
<ForkProcess(ForkPoolWorker-3, started daemon)> 9
<ForkProcess(ForkPoolWorker-5, started daemon)> 10
<ForkProcess(ForkPoolWorker-2, started daemon)> 11
<ForkProcess(ForkPoolWorker-1, started daemon)> 12
<ForkProcess(ForkPoolWorker-4, started daemon)> 13
<ForkProcess(ForkPoolWorker-3, started daemon)> 14
<ForkProcess(ForkPoolWorker-5, started daemon)> 15
<ForkProcess(ForkPoolWorker-2, started daemon)> 16
<ForkProcess(ForkPoolWorker-1, started daemon)> 17
<ForkProcess(ForkPoolWorker-3, started daemon)> 18
<ForkProcess(ForkPoolWorker-4, started daemon)> 19
EXCEPTION

但是如果我将池的进程数更改为等于或小于我机器上的内核数，则每次调用 f() where num甚至不打印。

使用 pool = multiprocessing.Pool(4) 输出:

<ForkProcess(ForkPoolWorker-1, started daemon)> 1
<ForkProcess(ForkPoolWorker-2, started daemon)> 3
<ForkProcess(ForkPoolWorker-3, started daemon)> 5
<ForkProcess(ForkPoolWorker-2, started daemon)> 7
<ForkProcess(ForkPoolWorker-1, started daemon)> 9
<ForkProcess(ForkPoolWorker-3, started daemon)> 11
<ForkProcess(ForkPoolWorker-3, started daemon)> 13
<ForkProcess(ForkPoolWorker-1, started daemon)> 15
<ForkProcess(ForkPoolWorker-2, started daemon)> 17
<ForkProcess(ForkPoolWorker-1, started daemon)> 19
EXCEPTION

我不明白为什么这些进程会被杀死，尤其是当函数中的 print 语句之后才抛出异常时。我真的不明白为什么只有当池中的进程数等于或小于机器上的内核数时才会发生这种情况。

Answer 1

引用multiprocessing.Pool.map的规范 你可以看到一个可选参数 chunksize，如果你将它指定为 1，即 pool.map(f, range(1,20), 1)，那么你会 yield预期的结果。

如果您增加 block 大小(例如 = 6)，您可能会看到:

<SpawnProcess(SpawnPoolWorker-1, started daemon)> 1
<SpawnProcess(SpawnPoolWorker-4, started daemon)> 7
<SpawnProcess(SpawnPoolWorker-3, started daemon)> 13
<SpawnProcess(SpawnPoolWorker-2, started daemon)> 19

这表明将 chunksize 的任务分配给 Pool 中的单个线程，当您在每个线程中引发异常时，当然不会执行剩余 chuck 中的任务。

从这里你可以知道 chunksize 的默认值在你的情况下是 2，这个变量存在的原因，很容易看出，是为了减少新线程的数量需要初始化(当您有适当的 block 大小时，这可能会节省资源和处理时间)。