python - 如何结合 multiprocessing 和 eventlet

Question

我有一个任务需要启动 2 个进程，并且在每个进程中需要启动 2 个线程才能真正工作。 下面是我用来模拟用例的源代码。

import multiprocessing
import eventlet

def subworker(num1, num2):
    print 'Start subworker %d,%d\n' % (num1, num2)
    eventlet.sleep(10)
    print 'End subworker %d,%d\n' % (num1, num2)

def worker(**kwargs):
    number = kwargs['number']
    pool = eventlet.GreenPool(size=2)
    pool.spawn_n(subworker, number, 1)
    pool.spawn_n(subworker, number, 2)
    pool.waitall()

def launcher(number):
    kwargs = {'number': number}
    th = multiprocessing.Process(target=worker, kwargs=kwargs)
    th.start()
    while True:
        if not th.is_alive():
            break
        eventlet.sleep(0)

    th.join()


def main():
    pool = eventlet.GreenPool(size=2)
    pool.spawn_n(launcher, 1)
    pool.spawn_n(launcher, 2)
    pool.waitall()

main()

当我运行这个 python 脚本时，我的预期输出是这样的:

Start subworker 1,1 
Start subworker 1,2
Start subworker 2,1
Start subworker 2,2
End subworker 1,1
End subworker 1,2
End subworker 2,1
End subworker 2,2

But what I really got is:

Start subworker 1,1
Traceback (most recent call last):

  File "/Users/leehom/python_local/lib/python2.7/site-packages/eventlet/greenpool.py", line 82, in _spawn_n_impl
Start subworker 1,2
    func(*args, **kwargs)

  File "/Users/leehom/Desktop/home/work_dir/source/snips/Test_multiprocessing_and_eventlet.py", line 27, in launcher
Start subworker 2,1
    if not th.is_alive():

  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 155, in is_alive
Start subworker 2,2
    assert self._parent_pid == os.getpid(), 'can only test a child process'

AssertionError: can only test a child process
Start subworker 1,1
Start subworker 1,2
Start subworker 2,1
Start subworker 2,2
End subworker 1,1
End subworker 1,2
End subworker 2,1
End subworker 2,2
End subworker 1,1
End subworker 1,2
End subworker 2,1
End subworker 2,2
Process finished with exit code 0

Seems launcher was called twice. I do not understand why this happen.

If I comment one line in my main function

def main():
    pool = eventlet.GreenPool(size=2)
    pool.spawn_n(launcher, 1)
    # pool.spawn_n(launcher, 2)
    pool.waitall()

结果是对的:

Start subworker 1,1
Start subworker 1,2
End subworker 1,1
End subworker 1,2

有人知道我该如何解决这个问题以及为什么会出现这个问题吗？

Answer 1

截至 2018-01，Eventlet 和多处理不能很好地协同工作。您最好的选择是在外部生成工作进程。第二个最佳选择是 os.fork() 创建工作进程然后才 import eventlet。

订阅此问题以在多处理兼容性得到解决时收到通知。 https://github.com/eventlet/eventlet/issues/147