我已经使用 sklearn 的 DecisionTreeClassifier
构建了一个文本分类模型,并想添加另一个预测器。我的数据位于 pandas 数据框中,列标记为 'Impression'
(文本)、'Volume'
( float )和 'Cancer'
(标签)。我一直只使用印象来预测癌症,但想改用印象和体积来预测癌症。
我之前运行的代码没有问题:
X_train, X_test, y_train, y_test = train_test_split(data['Impression'], data['Cancer'], test_size=0.2)
vectorizer = CountVectorizer()
X_train = vectorizer.fit_transform(X_train)
X_test = vectorizer.transform(X_test)
dt = DecisionTreeClassifier(class_weight='balanced', max_depth=6, min_samples_leaf=3, max_leaf_nodes=20)
dt.fit(X_train, y_train)
y_pred = dt.predict(X_test)
我尝试了几种不同的方法来添加音量预测器(更改以粗体显示):
1) 仅fit_transform
Impressions
X_train, X_test, y_train, y_test = train_test_split(data[['Impression', 'Volume']], data['Cancer'], test_size=0.2)
vectorizer = CountVectorizer()
X_train['Impression'] = vectorizer.fit_transform(X_train['Impression'])
X_test = vectorizer.transform(X_test)
dt = DecisionTreeClassifier(class_weight='balanced', max_depth=6, min_samples_leaf=3, max_leaf_nodes=20)
dt.fit(X_train, y_train)
y_pred = dt.predict(X_test)
这会引发错误
TypeError: float() argument must be a string or a number, not 'csr_matrix'
...
ValueError: setting an array element with a sequence.
2) 对 Impressions 和 Volumes 调用 fit_transform
。除了 fit_transform
行之外,与上面的代码相同:
X_train = vectorizer.fit_transform(X_train)
这当然会引发错误:
ValueError: Number of labels=1800 does not match number of samples=2
...
X_train.shape
(2, 2)
y_train.shape
(1800,)
我非常确定方法 #1 是正确的方法,但我找不到任何教程或解决方案来说明如何将 float 预测器添加到此文本分类模型。
如有任何帮助,我们将不胜感激!
最佳答案
ColumnTransformer()
将完全解决这个问题。我们可以在 ColumnTransformer
中将 remainder
参数设置为 passthrough
,而不是手动将 CountVectorizer
的输出附加到其他列>。
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.compose import make_column_transformer
from sklearn.feature_extraction.text import CountVectorizer
import pandas as pd
from sklearn import set_config
set_config(print_changed_only='True', display='diagram')
data = pd.DataFrame({'Impression': ['this is the first text',
'second one goes like this',
'third one is very short',
'This is the final statement'],
'Volume': [123, 1, 2, 123],
'Cancer': [1, 0, 0, 1]})
X_train, X_test, y_train, y_test = train_test_split(
data[['Impression', 'Volume']], data['Cancer'], test_size=0.5)
ct = make_column_transformer(
(CountVectorizer(), 'Impression'), remainder='passthrough')
pipeline = make_pipeline(ct, DecisionTreeClassifier())
pipeline.fit(X_train, y_train)
pipeline.score(X_test, y_test)
使用0.23.0版本,查看管道对象的视觉效果(set_config
中的display
参数)
关于python - 带有 CountVectorizer 和附加预测器的 sklearn DecisionTreeClassifier,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/61943972/