我使用 SciPy和 scikit-learn训练和应用多项式朴素贝叶斯分类器进行二进制文本分类。准确地说,我使用模块 sklearn.feature_extraction.text.CountVectorizer
用于从文本和模块 sklearn.naive_bayes.MultinomialNB
创建包含单词特征计数的稀疏矩阵作为分类器实现,用于在训练数据上训练分类器并将其应用于测试数据。
CountVectorizer
的输入是表示为 unicode 字符串的文本文档列表。训练数据远大于测试数据。我的代码看起来像这样(简化):
vectorizer = CountVectorizer(**kwargs)
# sparse matrix with training data
X_train = vectorizer.fit_transform(list_of_documents_for_training)
# vector holding target values (=classes, either -1 or 1) for training documents
# this vector has the same number of elements as the list of documents
y_train = numpy.array([1, 1, 1, -1, -1, 1, -1, -1, 1, 1, -1, -1, -1, ...])
# sparse matrix with test data
X_test = vectorizer.fit_transform(list_of_documents_for_testing)
# Training stage of NB classifier
classifier = MultinomialNB()
classifier.fit(X=X_train, y=y_train)
# Prediction of log probabilities on test data
X_log_proba = classifier.predict_log_proba(X_test)
问题:尽快MultinomialNB.predict_log_proba()
被调用,我得到 ValueError: dimension mismatch
。根据下面的IPython stacktrace,错误发生在SciPy中:
/path/to/my/code.pyc
--> 177 X_log_proba = classifier.predict_log_proba(X_test)
/.../sklearn/naive_bayes.pyc in predict_log_proba(self, X)
76 in the model, where classes are ordered arithmetically.
77 """
--> 78 jll = self._joint_log_likelihood(X)
79 # normalize by P(x) = P(f_1, ..., f_n)
80 log_prob_x = logsumexp(jll, axis=1)
/.../sklearn/naive_bayes.pyc in _joint_log_likelihood(self, X)
345 """Calculate the posterior log probability of the samples X"""
346 X = atleast2d_or_csr(X)
--> 347 return (safe_sparse_dot(X, self.feature_log_prob_.T)
348 + self.class_log_prior_)
349
/.../sklearn/utils/extmath.pyc in safe_sparse_dot(a, b, dense_output)
71 from scipy import sparse
72 if sparse.issparse(a) or sparse.issparse(b):
--> 73 ret = a * b
74 if dense_output and hasattr(ret, "toarray"):
75 ret = ret.toarray()
/.../scipy/sparse/base.pyc in __mul__(self, other)
276
277 if other.shape[0] != self.shape[1]:
--> 278 raise ValueError('dimension mismatch')
279
280 result = self._mul_multivector(np.asarray(other))
我不知道为什么会出现这个错误。有人可以向我解释一下并为这个问题提供解决方案吗?提前非常感谢!
最佳答案
听起来,就像您只需要对测试数据集使用 vectorizer.transform
一样,因为训练数据集固定了词汇表(毕竟您无法知道包括训练集在内的完整词汇表)。为了清楚起见,那是 vectorizer.transform
而不是 vectorizer.fit_transform
。
关于python - 值错误 : Dimension mismatch,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/12484310/