python - 使用 scikit-learn 分类到多个类别

Question

我正在尝试使用 scikit-learn 的一种监督学习方法将文本片段分类为一个或多个类别。我尝试过的所有算法的预测函数都只返回一个匹配项。

比如我有一段文字:

"Theaters in New York compared to those in London"

我已经训练算法为我输入的每个文本片段选择一个位置。

在上面的示例中，我希望它返回 New York 和 London，但它只返回 New York。

是否可以使用 scikit-learn 返回多个结果？或者甚至返回具有下一个最高概率的标签？

感谢您的帮助。

---更新

我尝试使用 OneVsRestClassifier，但我仍然只能获得每条文本的一个选项。下面是我正在使用的示例代码

y_train = ('New York','London')


train_set = ("new york nyc big apple", "london uk great britain")
vocab = {'new york' :0,'nyc':1,'big apple':2,'london' : 3, 'uk': 4, 'great britain' : 5}
count = CountVectorizer(analyzer=WordNGramAnalyzer(min_n=1, max_n=2),vocabulary=vocab)
test_set = ('nice day in nyc','london town','hello welcome to the big apple. enjoy it here and london too')

X_vectorized = count.transform(train_set).todense()
smatrix2  = count.transform(test_set).todense()


base_clf = MultinomialNB(alpha=1)

clf = OneVsRestClassifier(base_clf).fit(X_vectorized, y_train)
Y_pred = clf.predict(smatrix2)
print Y_pred

结果:['New York' 'London' 'London']

Answer 1

你想要的叫做多标签分类。 Scikits-learn 可以做到这一点。见这里:http://scikit-learn.org/dev/modules/multiclass.html .

我不确定您的示例中出了什么问题，我的 sklearn 版本显然没有 WordNGramAnalyzer。也许这是使用更多训练示例或尝试不同分类器的问题？但请注意，多标签分类器期望目标是元组列表/标签列表。

以下对我有用:

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.svm import LinearSVC
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.multiclass import OneVsRestClassifier

X_train = np.array(["new york is a hell of a town",
                    "new york was originally dutch",
                    "the big apple is great",
                    "new york is also called the big apple",
                    "nyc is nice",
                    "people abbreviate new york city as nyc",
                    "the capital of great britain is london",
                    "london is in the uk",
                    "london is in england",
                    "london is in great britain",
                    "it rains a lot in london",
                    "london hosts the british museum",
                    "new york is great and so is london",
                    "i like london better than new york"])
y_train = [[0],[0],[0],[0],[0],[0],[1],[1],[1],[1],[1],[1],[0,1],[0,1]]
X_test = np.array(['nice day in nyc',
                   'welcome to london',
                   'hello welcome to new york. enjoy it here and london too'])   
target_names = ['New York', 'London']

classifier = Pipeline([
    ('vectorizer', CountVectorizer(min_n=1,max_n=2)),
    ('tfidf', TfidfTransformer()),
    ('clf', OneVsRestClassifier(LinearSVC()))])
classifier.fit(X_train, y_train)
predicted = classifier.predict(X_test)
for item, labels in zip(X_test, predicted):
    print '%s => %s' % (item, ', '.join(target_names[x] for x in labels))

对我来说，这会产生输出:

nice day in nyc => New York
welcome to london => London
hello welcome to new york. enjoy it here and london too => New York, London

希望这会有所帮助。