我在尝试使用 multiprocessing.Pool.starmap 时遇到了一个奇怪的错误。重现错误所需的最少代码在这里:
from multiprocessing import Pool
# Ignore the fact that this class is useless as-is, it has more code but it wasn't relevant to the bug
class Coordinate(tuple) :
def __new__(cls, *args):
return tuple.__new__(cls, args)
#Essentially just stores two coordinates
class Move :
def __init__(self, oldPos, newPos) :
self.oldPos = oldPos
self.newPos = newPos
def __str__(self) :
return 'Old pos : ' + str(self.oldPos) + ' -- New pos : ' + str(self.newPos)
#Dummy function to show the problem
def funcThatNeedsTwoParams(move, otherParam) :
print(move)
# Second param ignored, no problem there
p = Pool(2)
moveOne = Move(Coordinate(0, 2), Coordinate(0, 1))
moveTwo = Move(Coordinate(2, 1), Coordinate(3, 0))
moveThree = Move(Coordinate(22345, -12400), Coordinate(153, 2357))
# The numbers are irrelevant, no effect on whether problem shows up or not
moves = [moveOne, moveTwo, moveThree]
paramsForStarmap = [[move, 'other param'] for move in moves]
print(paramsForStarmap)
#Output :
#[[<__main__.Move object at 0x1023d4438>, 'other param'], [<__main__.Move object at 0x1023d4470>, 'other param'], [<__main__.Move object at 0x1023d44a8>
for move in [params[0] for params in paramsForStarmap] :
print(move)
#Output :
#Old pos : (0, 2) -- New pos : (0, 1)
#Old pos : (2, 1) -- New pos : (3, 0)
#Old pos : (22345, -12400) -- New pos : (153, 2357)
p.starmap(funcThatNeedsTwoParams, paramsForStarmap)
#Output :
#Old pos : ((0, 2),) -- New pos : ((0, 1),)
#Old pos : ((22345, -12400),) -- New pos : ((153, 2357),)
#Old pos : ((2, 1),) -- New pos : ((3, 0),)
基本上,我有一个参数对数组,像这样:[[move, otherParam], [move, otherParam], ...],我打印出每个第一个参数以表明移动之前是有效的使用星图功能。然后我使用之前创建的池调用 starmap 函数,并告诉它使用我拥有的参数对。然后,莫名其妙地,每一步的坐标都变成了 ((coordinate), ) 形式的元组,而不是 (coordinate)。
我似乎无法弄清楚为什么 starmap 会更改传递给它的对象的属性,非常感谢任何帮助,谢谢。
最佳答案
这很有趣。问题不仅仅在于 starmap
。所有 Pool
函数都会发生这种情况 - apply
、map
等。事实证明,问题不在于 多重处理
。当您 pickle/unpickle Coordinate
类时会发生这种情况:
>>> c = Coordinate(0,2)
>>> print(c)
(0, 2)
>>> str(pickle.loads(pickle.dumps(c)))
'((0, 2),)'
Pickling tuple
子类并不像看起来那么简单,事实证明。您可以通过定义 __reduce__
来修复它修复酸洗过程的方法:
class Coordinate(tuple):
def __new__(cls, *args):
return tuple.__new__(cls, args)
def __reduce__(self):
return (self.__class__, tuple(self))
现在泡菜就好了:
>>> c = Coordinate(0,2)
>>> pickle.loads(pickle.dumps(c))
(0, 2)
而且您的示例代码也能正常工作。
关于python - Starmap 在传入参数之前修改参数?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/26259441/