python-3.x - 有没有办法使用python将不同的作业(进程)分配给linux中的特定核心？

Question

我有 python 函数，它们应该在使用多核的 linux 环境中并行运行。有没有办法明确指定每个进程应该使用哪个核心？

目前，我正在使用 python 多处理模块将这些 python 函数作为 4 核中的并行进程运行。

import multiprocessing as mp

def hello(name, msg):
    try:
        print("Hello {}".format(name))
        print(msg)
        return true
    except Exception:
        return False

pool = mp.Pool(mp.cpu_count())

msg = "It's a holiday!"
name_list = ["A", "B", "C"]

hello_status = pool.starmap(hello,[(name, msg) for name in name_list])

print(hello_status)

Answer 1

可能是 os.sched_setaffinity 和 os.sched_getaffinity 的组合。文档字符串说:

Signature: os.sched_setaffinity(pid, mask, /)

Docstring:

Set the CPU affinity of the process identified by pid to mask.

mask should be an iterable of integers identifying CPUs.

Type: builtin_function_or_method

我找不到特定于 Python 的文档，但是 these man pages应该是很好的第一个信息来源。

更新:

我决定研究 multiprocessing 模块并编写一个工作示例。我想出了两种使用 multiprocessing.Pool 类的方法。第一种方法是将 initializer 参数传递给 Pool 构造函数调用。第二种是使用 Pool.map 函数。

import os
from itertools import cycle
import multiprocessing as mp

def _myinit():
    my_pid = os.getpid()
    old_aff = os.sched_getaffinity(0)
    os.sched_setaffinity(0, [0, 3])
    new_aff = os.sched_getaffinity(0)
    print("My pid is {} and my old aff was {}, my new aff is {}".format(my_pid, old_aff, new_aff))
    
def map_hack(AFF):
    my_pid = os.getpid()
    old_aff = os.sched_getaffinity(0)
    os.sched_setaffinity(0, AFF)
    return (my_pid, old_aff, os.sched_getaffinity(0))

PROCESSES = os.cpu_count()

# just an example iterable you could use for the map_hack
# elements of cpus must be iterables, because of os.sched_setaffinity
_mycpus = cycle(os.sched_getaffinity(0))
cpus = [[next(_mycpus)] for x in range(PROCESSES)]

# Since Python 3.3 context managers are supported for mp.Pool

# using initializer argument to change affinity
with mp.Pool(processes=PROCESSES, initializer=_myinit) as pool:

    # do something conditional on your affinity 

    pool.close()
    pool.join()
    print("")

# using mp.Pool.map hack to change affinity
with mp.Pool(processes=PROCESSES) as pool:
    for x in pool.map(map_hack, cpus, chunksize=1):
        print("My pid is {} and my old aff was {}, my new aff is {}".format(*x))

    # do something conditional on your affinity 

    pool.close()
    pool.join()

请注意，使用 initializer 我硬编码了第一个和第四个 CPU (0, 3) 的所有进程的亲和性，但这只是因为我发现使用 cycle 就像我对 map_hack 所做的一样。我还想证明您可以为任何(合法)数量的 CPU 设置关联。

我建议您通读代码并确保通过阅读相关文档并通过更改一些参数来玩弄它来理解它。不用说，所有的 print 语句只是为了让我们自己相信这些方法有效。

最后，如果您需要更多控制权，我建议您使用 mp.Process 对象而不是 mp.Pool。 os 中的相同工具也应该在那里派上用场。

window :

如果您使用的是 Windows，这将不起作用。来自docs :

These functions control how a process is allocated CPU time by the operating system. They are only available on some Unix platforms. For more detailed information, consult your Unix manpages.

在这种情况下，您可以查看 win32process，特别是 win32process.SetProcessAffinityMask 和 win32process.GetProcessAffinityMask，请参阅 here .