我想使用 Python 多处理来运行网格搜索以查找预测模型。 当我查看核心使用情况时,它似乎总是只使用一个核心。知道我做错了什么吗?
import multiprocessing
from sklearn import svm
import itertools
#first read some data
#X will be my feature Numpy 2D array
#y will be my 1D Numpy array of labels
#define the grid
C = [0.1, 1]
gamma = [0.0]
params = [C, gamma]
grid = list(itertools.product(*params))
GRID_hx = []
def worker(par, grid_list):
#define a sklearn model
clf = svm.SVC(C=g[0], gamma=g[1],probability=True,random_state=SEED)
#run a cross validation function: returns error
ll = my_cross_validation_function(X, y, model=clf, n=1, test_size=0.2)
print(par, ll)
grid_list.append((par, ll))
if __name__ == '__main__':
manager = multiprocessing.Manager()
GRID_hx = manager.list()
jobs = []
for g in grid:
p = multiprocessing.Process(target=worker, args=(g,GRID_hx))
jobs.append(p)
p.start()
p.join()
print("\n-------------------")
print("SORTED LIST")
print("-------------------")
L = sorted(GRID_hx, key=itemgetter(1))
for l in L[:5]:
print l
最佳答案
您的问题是您加入每项工作在您开始后立即:
for g in grid:
p = multiprocessing.Process(target=worker, args=(g,GRID_hx))
jobs.append(p)
p.start()
p.join()
join 阻塞,直到相应的进程完成工作。这意味着您的代码一次只启动一个进程,等待它完成,然后再启动下一个进程。
为了让所有进程并行运行,您需要首先将它们全部启动,然后将它们全部加入:
jobs = []
for g in grid:
p = multiprocessing.Process(target=worker, args=(g,GRID_hx))
jobs.append(p)
p.start()
for j in jobs:
j.join()
文档:link
关于Python 多处理似乎不使用多个内核,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/29802503/