python - 在子进程中使用多处理

Question

在Windows中，在使用多处理之前必须检查进程是否是ma​​in，否则会出现无限循环。

我尝试将进程的名称更改为子进程的名称，以在我调用的类或函数中使用多重处理，但没有成功。这可能吗？到目前为止，我无法使用多处理，除非它使用主进程。

如果可能的话，有人可以提供一个示例，说明如何在从更高进程调用的类或函数中使用多重处理吗？谢谢。

编辑:

这是一个示例 - 第一个有效，但所有内容都在 1 个文件中完成: simplemtexample3.py:

import random
import multiprocessing
import math

def mp_factorizer(nums, nprocs):
    #schtze den prozess
    #print __name__
    if __name__ == '__main__':
        out_q = multiprocessing.Queue()
        chunksize = int(math.ceil(len(nums) / float(nprocs)))
        procs = []
        for i in range(nprocs):

            p = multiprocessing.Process(
                    target=worker,            
                    args=(nums[chunksize * i:chunksize * (i + 1)],
                          out_q))
            procs.append(p)
            p.start()

        # Collect all results into a single result dict. We know how many dicts
        # with results to expect.
        resultlist = []
        for i in range(nprocs):
            temp=out_q.get()
            index =0
            #print temp
            for i in temp:
                resultlist.append(temp[index][0][0:])
                index +=1

        # Wait for all worker processes to finish
        for p in procs:
            p.join()
            resultlist2 = [x for x in resultlist if x != []]
        return resultlist2

def worker(nums, out_q):
    """ The worker function, invoked in a process. 'nums' is a
        list of numbers to factor. The results are placed in
        a dictionary that's pushed to a queue.
    """
    outlist = []

    for n in nums:
        newnumber= n*2
        newnumberasstring = str(newnumber)
        if newnumber:
            outlist.append(newnumberasstring)
    out_q.put(outlist)

l = []
for i in range(80):
    l.append(random.randint(1,8))

print mp_factorizer(l, 4)

但是，当我尝试从另一个文件调用 mp_factorizer 时，它不起作用，因为 if __name__ == '__main__':

simpletexample.py

import random
import multiprocessing
import math

def mp_factorizer(nums, nprocs):
    #schtze den prozess
    #print __name__
    if __name__ == '__main__':
        out_q = multiprocessing.Queue()
        chunksize = int(math.ceil(len(nums) / float(nprocs)))
        procs = []
        for i in range(nprocs):

            p = multiprocessing.Process(
                    target=worker,            
                    args=(nums[chunksize * i:chunksize * (i + 1)],
                          out_q))
            procs.append(p)
            p.start()

        # Collect all results into a single result dict. We know how many dicts
        # with results to expect.
        resultlist = []
        for i in range(nprocs):
            temp=out_q.get()
            index =0
            #print temp
            for i in temp:
                resultlist.append(temp[index][0][0:])
                index +=1

        # Wait for all worker processes to finish
        for p in procs:
            p.join()
            resultlist2 = [x for x in resultlist if x != []]
        return resultlist2

def worker(nums, out_q):
    """ The worker function, invoked in a process. 'nums' is a
        list of numbers to factor. The results are placed in
        a dictionary that's pushed to a queue.
    """
    outlist = []

    for n in nums:
        newnumber= n*2
        newnumberasstring = str(newnumber)
        if newnumber:
            outlist.append(newnumberasstring)
    out_q.put(outlist)

startsimpletexample.py

import simplemtexample as smt
import random

l = []
for i in range(80):
    l.append(random.randint(1,8))

print smt.mp_factorizer(l, 4)

Answer 1

if __name__ == '__main__'是强制性的(至少在Windows中)，如果想使用多处理。

在 Windows 中，它的工作原理如下:对于要生成的每个工作线程，Windows 将自动启动主进程，并再次启动所有需要的文件。但是，只有第一个已启动的进程称为ma​​in。这就是为什么使用 if __name__ == '__main__' 阻止 mt_factorizer 的执行可以防止多处理创建无限循环。

所以本质上Windows需要读取包含worker的文件，以及worker调用的所有函数——对于每个worker。通过阻止 mt_factorizer，我们确保不会创建额外的工作线程，而窗口仍然可以执行工作线程。这就是为什么将所有代码都放在一个文件中的多处理示例会直接阻止创建工作线程(就像本例中的 mt_factorizer 所做的那样)(但不会阻止工作函数)，因此 Windows 仍然可以执行工作函数。如果所有代码都在一个文件中，并且整个文件都受到保护，则无法创建任何工作程序。

如果多处理代码位于另一个类中并被调用，则需要直接在调用上方实现if __name__ == '__main__': mpteststart.py

import random
import mptest as smt

l = []
for i in range(4):
    l.append(random.randint(1,8))
print "Random numbers generated"
if __name__ == '__main__':
    print smt.mp_factorizer(l, 4)

mptest.py

import multiprocessing
import math

print "Reading mptest.py file"
def mp_factorizer(nums, nprocs):

    out_q = multiprocessing.Queue()
    chunksize = int(math.ceil(len(nums) / float(nprocs)))
    procs = []
    for i in range(nprocs):

        p = multiprocessing.Process(
                target=worker,            
                args=(nums[chunksize * i:chunksize * (i + 1)],
                      out_q))
        procs.append(p)
        p.start()

    # Collect all results into a single result dict. We know how many dicts
    # with results to expect.
    resultlist = []
    for i in range(nprocs):
        temp=out_q.get()
        index =0
        #print temp
        for i in temp:
            resultlist.append(temp[index][0][0:])
            index +=1

    # Wait for all worker processes to finish
    for p in procs:
        p.join()
        resultlist2 = [x for x in resultlist if x != []]
    return resultlist2

def worker(nums, out_q):
    """ The worker function, invoked in a process. 'nums' is a
        list of numbers to factor. The results are placed in
        a dictionary that's pushed to a queue.
    """
    outlist = []

    for n in nums:
        newnumber= n*2
        newnumberasstring = str(newnumber)
        if newnumber:
            outlist.append(newnumberasstring)
    out_q.put(outlist)

在上面的代码中，if __name__ == '__main__' 已被删除，因为它已经在调用文件中。

然而，结果却有些出乎意料:

Reading mptest.py file
random numbers generated
Reading mptest.py file
random numbers generated
worker started
Reading mptest.py file
random numbers generated
worker started
Reading mptest.py file
random numbers generated
worker started
Reading mptest.py file
random numbers generated
worker started
['1', '1', '4', '1']

多处理被阻止无限执行，但其余代码仍被执行多次(在本例中为随机数生成)。这不仅会导致性能下降，还可能导致其他严重的错误。解决方案是，如果在某处使用多处理，则保护整个主进程不被 Windows 重复执行: mptest.py

import random
import mptest as smt

if __name__ == '__main__':  
    l = []
    for i in range(4):
        l.append(random.randint(1,8))
    print "random numbers generated"   
    print smt.mp_factorizer(l, 4)

现在我们得到的只是期望的结果，随机数只生成一次:

Reading mptest.py file
random numbers generated
Reading mptest.py file
worker started
Reading mptest.py file
worker started
Reading mptest.py file
worker started
Reading mptest.py file
worker started
['1', '6', '2', '1']

请注意，在本例中，mpteststart.py 是主进程。如果不是，if __name__ == '__main__' 必须在调用链中向上移动，直到它位于主进程中。 一旦主进程受到这种方式的保护，就不会再出现不必要的重复代码执行。