python - 为什么 multiprocessing.Pool.map 比内置 map 慢？

Question

import multiprocessing
import time
from subprocess import call,STDOUT
from glob import glob
import sys


def do_calculation(data):
    x = time.time()
    with open(data + '.classes.report','w') as f:
        call(["external script", data], stdout = f.fileno(), stderr=STDOUT)
    return 'apk: {data!s} time {tim!s}'.format(data = data ,tim = time.time()-x)


def start_process():
    print 'Starting', multiprocessing.current_process().name

if __name__ == '__main__':

    inputs = glob('./*.dex')


    builtin_outputs = map(do_calculation, inputs)
    print 'Built-in:'
    for i in builtin_outputs:
        print i

    pool_size = multiprocessing.cpu_count() * 2
    print 'Worker Pool size: %s' % pool_size
    pool = multiprocessing.Pool(processes=pool_size,
                                initializer=start_process,
                                )
    pool_outputs = pool.map(do_calculation, inputs)
    pool.close() # no more tasks
    pool.join()  # wrap up current tasks

    print 'Pool output:'
    for i in pool_outputs:
        print i

令人惊讶的是，builtin_outputs 的执行时间比pool_outputs 更快:

Built-in:
apk: ./TooDo_2.0.8.classes.dex time 5.69289898872
apk: ./TooDo_2.0.9.classes.dex time 5.37206411362
apk: ./Twitter_Client.classes.dex time 0.272782087326
apk: ./zaTelnet_Light.classes.dex time 0.141801118851
apk: ./Temperature_Converter.classes.dex time 0.270312070847
apk: ./Tipper_1.0.classes.dex time 0.293262958527
apk: ./XLive.classes.dex time 0.361288070679
apk: ./TwitterDroid_0.1.2_alpha.classes.dex time 0.381947040558
apk: ./Universal_Conversion_Application.classes.dex time 0.404763936996

Worker Pool size: 8

Pool output:
apk: ./TooDo_2.0.8.classes.dex time 5.72440505028
apk: ./TooDo_2.0.9.classes.dex time 5.9017829895
apk: ./Twitter_Client.classes.dex time 0.309305906296
apk: ./zaTelnet_Light.classes.dex time 0.374011039734
apk: ./Temperature_Converter.classes.dex time 0.450366973877
apk: ./Tipper_1.0.classes.dex time 0.379780054092
apk: ./XLive.classes.dex time 0.394504070282
apk: ./TwitterDroid_0.1.2_alpha.classes.dex time 0.505702018738
apk: ./Universal_Conversion_Application.classes.dex time 0.512043952942

如何解释这种性能差异？

Answer 1

当您使用多进程时，您有必要为工作进程提供足够的计算以持续至少几秒钟。如果工作进程结束得太快，那么就会花费太多时间来设置池、生成子进程以及(可能)在进程之间切换(并且没有足够的时间实际进行预期的计算)以证明使用 multiprocessing。

此外，如果您的计算受 CPU 限制，则初始化进程数多于内核数的池 (multiprocessing.cpu_count()) 会适得其反。它将使操作系统在进程之间切换，同时不允许计算进行得更快。