我有一个函数来计算特征,然后将特征保存到 pickle 中。
test_knn_feats = NNF.predict(X_test)
np.save('data/knn_feats_%s_test.npy' % metric , test_knn_feats)
在函数中,如果n_jobs
大于1,则将执行下面的代码。
fest_feats =[]
pool = Pool(processes = self.n_jobs)
for i in range(X.shape[0]):
test_feats.append(pool.apply_async(self.get_features_for_one(X[i:i+1])))
pool.close()
pool.join()
return np.vstack(test_feats)
但是,出现错误:
TypeError Traceback (most recent call last)
<ipython-input-96-4f707b7cd533> in <module>()
12 print(test_knn_feats)
13 # Dump the features to disk
---> 14 np.save('data/knn_feats_%s_test.npy' % metric , test_knn_feats)
/opt/conda/lib/python3.6/site-packages/numpy/lib/npyio.py in save(file, arr, allow_pickle, fix_imports)
507 arr = np.asanyarray(arr)
508 format.write_array(fid, arr, allow_pickle=allow_pickle,
--> 509 pickle_kwargs=pickle_kwargs)
510 finally:
511 if own_fid:
/opt/conda/lib/python3.6/site-packages/numpy/lib/format.py in write_array(fp, array, version, allow_pickle, pickle_kwargs)
574 if pickle_kwargs is None:
575 pickle_kwargs = {}
--> 576 pickle.dump(array, fp, protocol=2, **pickle_kwargs)
577 elif array.flags.f_contiguous and not array.flags.c_contiguous:
578 if isfileobj(fp):
函数get_features_for_one
将返回一个列表,如下所示。
...
knn_feats = np.hstack(return_list)
assert knn_feats.shape == (239,) or knn_feats.shape == (239, 1)
return knn_feats
*更新:
test_feats =[]
pool = Pool(processes = self.n_jobs)
for i in range(X.shape[0]):
test_feats.append(pool.apply_async(self.get_features_for_one, (X[i:i+1],)))
test_feats= [res.get() for res in test_feats]
pool.close()
pool.join()
return np.vstack(test_feats)
最佳答案
这里有两个主要错误:
test_feats =[] # you called it fest_feats, I assume a typo
pool = Pool(processes = self.n_jobs)
for i in range(X.shape[0]):
test_feats.append(pool.apply_async(self.get_features_for_one(X[i:i+1])))
pool.close()
pool.join()
return np.vstack(test_feats)
首先,您创建一个池。然后,对于每个
i
,您提交一项作业,然后关闭并加入池中。您应该只在最后、在循环之外关闭并加入池一次。test_feats 最终成为“ future ”列表,而不是实际数据。所以 vstack() 对它们来说是没有意义的。您需要对每个 future 调用
get()
来获取get_features_for_one()
的结果,然后将该列表传递给vstack()
。例如np.vstack([res.get() for res in test_feats])
。
简而言之,你的问题与你最终从 numpy.save() 收到的 TypeError 无关——你的问题是你的逻辑完全被破坏了,你的数据不是你想要的。认为是这样。
关于python - numpy.save() 内的 Pickle TypeError,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/47958111/