我正在尝试 pickle 一个 method_descriptor
。
使用 pickle
或 cloudpickle
进行 pickle 失败:
Python 2.7.10 |Continuum Analytics, Inc.| (default, Oct 19 2015, 18:04:42)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import pickle, cloudpickle
>>> pickle.dumps(set.union)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 1374, in dumps
Pickler(file, protocol).dump(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/pickle.py", line 306, in save
rv = reduce(self.proto)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: cannot pickle method_descriptor objects
>>> cloudpickle.dumps(set.union)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 602, in dumps
cp.dump(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 111, in dump
raise pickle.PicklingError(msg)
pickle.PicklingError: Could not pickle object as excessively deep recursion required.
导入 dill
以某种方式使 pickle
工作,如下所示:
>>> import dill
>>> pickle.dumps(set.union)
'cdill.dill\n_getattr\np0\n(c__builtin__\nset\np1\nS\'union\'\np2\nS"<method \'union\' of \'set\' objects>"\np3\ntp4\nRp5\n.'
>>> f = pickle.loads(pickle.dumps(set.union))
>>> set.union(set([1,2]), set([3]))
set([1, 2, 3])
>>> f(set([1,2]), set([3]))
set([1, 2, 3])
即使在 dill
导入之后,cloudpickle
中的问题仍然存在:
>>> cloudpickle.dumps(set.union)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 602, in dumps
cp.dump(obj)
File "/home/pmd/anaconda3/envs/python2/lib/python2.7/site-packages/cloudpickle/cloudpickle.py", line 111, in dump
raise pickle.PicklingError(msg)
pickle.PicklingError: Could not pickle object as excessively deep recursion required.
在我的应用程序中,我依靠 cloudpickle
来处理具有全局变量的函数。所以我的问题是,如何让 cloudpickle
为 Python 2.7 中的 method_descriptor
对象工作?
编辑:我注意到同样的问题出现在 Python 3.3 中,但不存在于 Python 3.5 中。
最佳答案
我是 dill
的作者。当您执行 import dill
时,它将序列化注册表从 dill
注入(inject)到 pickle
(基本上,将所有 copy_reg
-将 dill
中的知识输入到 pickle
注册表中)。
>>> import pickle
>>> pickle.Pickler.dispatch
{<type 'function'>: <function save_global at 0x105d0c7d0>, <type 'dict'>: <function save_dict at 0x105d0c668>, <type 'int'>: <function save_int at 0x105d0c230>, <type 'long'>: <function save_long at 0x105d0c2a8>, <type 'list'>: <function save_list at 0x105d0c578>, <type 'str'>: <function save_string at 0x105d0c398>, <type 'unicode'>: <function save_unicode at 0x105d0c410>, <type 'instance'>: <function save_inst at 0x105d0c758>, <type 'type'>: <function save_global at 0x105d0c7d0>, <type 'NoneType'>: <function save_none at 0x105d0c140>, <type 'bool'>: <function save_bool at 0x105d0c1b8>, <type 'tuple'>: <function save_tuple at 0x105d0c488>, <type 'float'>: <function save_float at 0x105d0c320>, <type 'classobj'>: <function save_global at 0x105d0c7d0>, <type 'builtin_function_or_method'>: <function save_global at 0x105d0c7d0>}
>>> import dill
>>> pickle.Pickler.dispatch
{<class '_pyio.BufferedReader'>: <function save_file at 0x106c8b848>, <class '_pyio.TextIOWrapper'>: <function save_file at 0x106c8b848>, <type 'operator.itemgetter'>: <function save_itemgetter at 0x106c8b578>, <type 'weakproxy'>: <function save_weakproxy at 0x106c8c050>, <type 'NoneType'>: <function save_none at 0x105d0c140>, <type 'str'>: <function save_string at 0x105d0c398>, <type 'file'>: <function save_file at 0x106c8b8c0>, <type 'classmethod'>: <function save_classmethod at 0x106c8c230>, <type 'float'>: <function save_float at 0x105d0c320>, <type 'instancemethod'>: <function save_instancemethod0 at 0x106c8ba28>, <type 'cell'>: <function save_cell at 0x106c8bb18>, <type 'member_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'slice'>: <function save_slice at 0x106c8bc80>, <type 'dict'>: <function save_module_dict at 0x106c8b410>, <type 'long'>: <function save_long at 0x105d0c2a8>, <type 'code'>: <function save_code at 0x106c8b320>, <type 'type'>: <function save_type at 0x106c8c0c8>, <type 'xrange'>: <function save_singleton at 0x106c8bde8>, <type 'builtin_function_or_method'>: <function save_builtin_method at 0x106c8b9b0>, <type 'classobj'>: <function save_classobj at 0x106c8b488>, <type 'weakref'>: <function save_weakref at 0x106c8bed8>, <type 'getset_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'weakcallableproxy'>: <function save_weakproxy at 0x106c8c050>, <class '_pyio.BufferedRandom'>: <function save_file at 0x106c8b848>, <type 'int'>: <function save_int at 0x105d0c230>, <type 'list'>: <function save_list at 0x105d0c578>, <type 'functools.partial'>: <function save_functor at 0x106c8b7d0>, <type 'bool'>: <function save_bool at 0x105d0c1b8>, <type 'function'>: <function save_function at 0x106c8b398>, <type 'thread.lock'>: <function save_lock at 0x106c8b500>, <type 'super'>: <function save_functor at 0x106c8b938>, <type 'staticmethod'>: <function save_classmethod at 0x106c8c230>, <type 'module'>: <function save_module at 0x106c8bf50>, <type 'method_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'operator.attrgetter'>: <function save_attrgetter at 0x106c8b5f0>, <type 'wrapper_descriptor'>: <function save_wrapper_descriptor at 0x106c8bc08>, <type 'numpy.ufunc'>: <function save_numpy_ufunc at 0x106c8bcf8>, <type 'method-wrapper'>: <function save_instancemethod at 0x106c8baa0>, <type 'instance'>: <function save_inst at 0x105d0c758>, <type 'cStringIO.StringI'>: <function save_stringi at 0x106c8b6e0>, <type 'unicode'>: <function save_unicode at 0x105d0c410>, <class '_pyio.BufferedWriter'>: <function save_file at 0x106c8b848>, <type 'property'>: <function save_property at 0x106c8c140>, <type 'ellipsis'>: <function save_singleton at 0x106c8bde8>, <type 'tuple'>: <function save_tuple at 0x105d0c488>, <type 'cStringIO.StringO'>: <function save_stringo at 0x106c8b758>, <type 'NotImplementedType'>: <function save_singleton at 0x106c8bde8>, <type 'dictproxy'>: <function save_dictproxy at 0x106c8bb90>}
cloudpickle
与 dill
具有(略微)不同的 pickle 函数,如果您使用 cloudpickle
,它会将自己的序列化函数插入pickle
注册表。如果你想让 cloudpickle
为你工作,你也许可以通过 monkeypatch 解决方案......本质上是在你的应用程序中安装一个模块,它可以 import dill as cloudpickle
(很好的引用: http://blog.dscpl.com.au/2015/03/safely-applying-monkey-patches-in-python.html )... 但这将在您的应用程序上下文中将 cloudpickle
的整个使用替换为 dill
。您也可以按照以下方式尝试使用 monkeypatch:
>>> #first import dill, which populates itself into pickle's dispatch
>>> import dill
>>> import pickle
>>> # save the MethodDescriptorType from dill
>>> MethodDescriptorType = type(type.__dict__['mro'])
>>> MethodDescriptorWrapper = pickle.Pickler.dispatch[MethodDescriptorType]
>>> # cloudpickle does the same, so let it update the dispatch table
>>> import cloudpickle
>>> # now, put the saved MethodDescriptorType back in
>>> pickle.Pickler.dispatch[MethodDescriptorWrapperType] = MethodDescriptorWrapper
请注意,如果您要直接使用 cloudpickle.dumps
,则必须通过在 上执行上述 monkeypatch 直接重载
.cloudpickle
中的注册表cloudpickle.CloudPickler.dispatch
我不保证它会工作,也不保证它不会搞砸 cloudpickle
中的其他对象(基本上,我还没有尝试过它),但这是用 dill
中的包装器替换有问题的 cloudpickle
包装器的潜在途径。
如果您想要简短的答案,我会说(至少对于这种情况)使用 dill
。 ;)
编辑 copyreg
:
这是dill
中的内容:
def _getattr(objclass, name, repr_str):
# hack to grab the reference directly
try:
attr = repr_str.split("'")[3]
return eval(attr+'.__dict__["'+name+'"]')
except:
attr = getattr(objclass,name)
if name == '__dict__':
attr = attr[name]
return attr
用于注册具有较低级别 reduce 函数的函数(直接在 pickler 实例上)。 obj
是要 pickle 的对象。
pickler.save_reduce(_getattr, (obj.__objclass__, obj.__name__, obj.__repr__()), obj=obj)
我相信这会转化为像这样的 reduce 方法(直接在 copyreg.pickle
中使用):
def _reduce_method_descriptor(obj):
return _getattr, (obj.__objclass__, obj.__name__, obj.__repr__())
关于python - python中的 pickle 方法描述符对象,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/34124270/