python - 对标记文本进行分类时出现问题，预测错误？

Question

我正在使用 scikit-learn 提供的不同分类器和矢量化器，所以假设我有以下内容:

training = [["this was a good movie, 'POS'"],
      ["this was a bad movie, 'NEG'"],
      ["i went to the movies, 'NEU'"], 
      ["this movie was very exiting it was great, 'POS'"], 
      ["this is a boring film, 'NEG'"]
        ,........................,
          [" N-sentence, 'LABEL'"]]

#Where each element of the list is another list that have documents, then.

splitted = [#remove the tags from training]

from sklearn.feature_extraction.text import HashingVectorizer
X = HashingVectorizer(
    tokenizer=lambda  doc: doc, lowercase=False).fit_transform(splitted)

print X.toarray()

然后我有这个向量表示:

[[ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]
 [ 0.  0.  0. ...,  0.  0.  0.]]

这样做的问题是我不知道我是否对语料库进行了正确的矢量化，然后:

#This is the test corpus:
test = ["I don't like this movie it sucks it doesn't liked me"]

#I vectorize the corpus with hashing vectorizer
Y = HashingVectorizer(
    tokenizer=lambda  doc: doc, lowercase=False).fit_transform(test)

然后我打印Y:

[[ 0.  0.  0. ...,  0.  0.  0.]]

然后

y = [x[-1]for x in training]

#import SVM and classify
from sklearn.svm import SVC
svm = SVC()
svm.fit(X, y)
result = svm.predict(X)
print "\nThe opinion is:\n",result

问题是，我得到了以下[NEG]，这实际上是正确的预测:

["this was a good movie, 'POS'"]

我想我没有对训练进行矢量化，或者y目标是错误的，任何人都可以帮助我了解发生了什么以及我应该如何对训练进行矢量化 为了有正确的预测？

Answer 1

我将让您将训练数据转换为预期格式:

training = ["this was a good movie",
            "this was a bad movie",
            "i went to the movies",
            "this movie was very exiting it was great", 
            "this is a boring film"]

labels = ['POS', 'NEG', 'NEU', 'POS', 'NEG']

特征提取

>>> from sklearn.feature_extraction.text import HashingVectorizer
>>> vect = HashingVectorizer(n_features=5, stop_words='english', non_negative=True)
>>> X_train = vect.fit_transform(training)
>>> X_train.toarray()
[[ 0.          0.70710678  0.          0.          0.70710678]
 [ 0.70710678  0.70710678  0.          0.          0.        ]
 [ 0.          0.          0.          0.          0.        ]
 [ 0.          0.89442719  0.          0.4472136   0.        ]
 [ 1.          0.          0.          0.          0.        ]]

对于更大的语料库，您应该增加 n_features 以避免冲突，我使用了 5 个，以便可以可视化结果矩阵。另请注意，我使用了 stop_words='english'，我认为示例如此之少，删除停用词很重要，否则您可能会混淆分类器。

模型训练

from sklearn.svm import SVC

model = SVC()
model.fit(X_train, labels)

预测

>>> test = ["I don't like this movie it sucks it doesn't liked me"]
>>> X_pred = vect.transform(test)
>>> model.predict(X_pred)
['NEG']

>>> test = ["I think it was a good movie"]
>>> X_pred = vect.transform(test)
>>> model.predict(X_pred)
['POS']

编辑:请注意，第一个测试示例的正确分类只是一个幸运的巧合，因为我没有看到任何可以从训练集中学习的单词是否定的。在第二个示例中，单词good 可能会触发积极分类。