python - 从 Scikit Learn 中保存的训练分类器做出预测

Question

我用 Python 为推文编写了一个分类器，然后以 .pkl 格式将其保存在磁盘上，这样我就可以一次又一次地运行它，而无需每次都对其进行训练。这是代码:

import pandas
import re
from sklearn.feature_extraction import FeatureHasher

from sklearn.feature_selection import SelectKBest
from sklearn.feature_selection import chi2

from sklearn import cross_validation

from sklearn.externals import joblib


#read the dataset of tweets

header_row=['sentiment','tweetid','date','query', 'user', 'text']
train = pandas.read_csv("training.data.csv",names=header_row)

#keep only the right columns

train = train[["sentiment","text"]]

#remove puctuation, special characters, numbers and lower case the text

def remove_spch(text):

    return re.sub("[^a-z]", ' ', text.lower())

train['text'] = train['text'].apply(remove_spch)


#Feature Hashing

def tokens(doc):
    """Extract tokens from doc.

    This uses a simple regex to break strings into tokens.
    """
    return (tok.lower() for tok in re.findall(r"\w+", doc))

n_features = 2**18
hasher = FeatureHasher(n_features=n_features, input_type="string", non_negative=True)
X = hasher.transform(tokens(d) for d in train['text'])

y = train['sentiment']

X_new = SelectKBest(chi2, k=20000).fit_transform(X, y)

a_train, a_test, b_train, b_test = cross_validation.train_test_split(X_new, y, test_size=0.2, random_state=42)

from sklearn.ensemble import RandomForestClassifier 

classifier=RandomForestClassifier(n_estimators=10)                  
classifier.fit(a_train.toarray(), b_train)                            
prediction = classifier.predict(a_test.toarray()) 

#Export the trained model to load it in another project

joblib.dump(classifier, 'my_model.pkl', compress=9)

假设我有另一个 Python 文件，我想对推文进行分类。如何进行分类？

from sklearn.externals import joblib
model_clone = joblib.load('my_model.pkl')

mytweet = 'Uh wow:@medium is doing a crowdsourced data-driven investigation tracking down a disappeared refugee boat'

在 hasher.transform 之前，我可以复制相同的过程以将其添加到预测模型中，但是我遇到了无法计算最佳 20k 特征的问题。要使用 SelectKBest，您需要同时添加特征和标签。因为我想预测标签，所以我不能使用 SelectKBest。那么，我怎样才能通过这个问题继续进行预测呢？

Answer 1

我支持@EdChum 的评论

you build a model by training it on data which presumably is representative enough for it to cope with unseen data

实际上这意味着您需要将 FeatureHasher 和 SelectKBest 应用到您的新数据中 predict 仅 . (在新数据上重新训练 FeatureHasher 是错误的，因为通常它会产生不同的特征)。

要么这样做

• pickle FeatureHasher 和 SelectKBest 分别

或(更好)

• 创建一个包含 FeatureHasher、SelectKBest 和 RandomForestClassifier 的管道，并对整个管道进行 pickle。然后您可以加载此管道并对新数据使用 predict。