python - 如何在 scikit learn 中为 cross_validate 制定自定义评分指标？

Question

我想开发 pr_auc 作为 cross_validate() 的评分指标。所以我遵循了 Scikit Learn 的用户指南:https://scikit-learn.org/stable/modules/model_evaluation.html#scoring

我的代码如下所示:

from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate
from xgboost import XGBClassifier
from sklearn.metrics import auc, make_scorer

def cus_pr_auc(x, y):
    score=auc(x, y)
    return score

X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
    n_clusters_per_class=2, weights=[0.9], flip_y=0, random_state=7)

model = XGBClassifier(scale_pos_weight=9)

scores = cross_validate(model, X, y, scoring=make_scorer(cus_pr_auc, greater_is_better=True), cv=3, n_jobs=-1)

但是，我收到以下错误消息:

ValueError: x is neither increasing nor decreasing : [1 1 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 1 0 0 1 0 0 0 0 1 1 0 0 0 1 0 0 0 0 0 0 0 1 0 0 1].

如何修复代码？

Answer 1

将您的指标从 auc 更改为 roc_auc_score，如下所示，即可开始:

from sklearn.datasets import make_classification
from sklearn.model_selection import cross_validate
from xgboost import XGBClassifier
from sklearn.metrics import roc_auc_score, make_scorer

def cus_pr_auc(x, y):
    score=roc_auc_score(x, y)
    return score

X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
    n_clusters_per_class=2, weights=[0.9], flip_y=0, random_state=7)

model = XGBClassifier(scale_pos_weight=9)

scores = cross_validate(model, X, y, scoring=make_scorer(cus_pr_auc, greater_is_better=True), cv=3, n_jobs=-1)

scores
{'fit_time': array([0.0627017, 0.0569284, 0.046772 ]),
 'score_time': array([0.00534487, 0.00616908, 0.00347471]),
 'test_score': array([0.90244639, 0.90242424, 0.94969697])}

您的 auc 无法正常工作，因为根据docs :

x coordinates. These must be either monotonic increasing or monotonic decreasing.

因此出现了错误(另请参阅 here，了解错误消息的具体来源)。

通常，auc 是一个低级评分指标，用作 auc(fpr,tpr)。