arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2209.02064stat.MEcs.LGmath.STstat.MLstat.TH
收藏

GRASP:分类学习的拟合优度检验

GRASP: A Goodness-of-Fit Test for Classification Learning

University of Southern California(南加州大学)

Adel Javanmard, Mohammad Mehrabi

AI总结:

本文提出GRASP检验,用于评估二元分类器对条件标签分布的拟合优度,无需参数假设,通过容差假设检验和f散度度量,并扩展了model-X版本以提升功效。

摘要:

分类器的性能通常以测试数据上的平均准确率来衡量。尽管平均准确率是标准度量,但它无法刻画模型对给定特征向量下标签条件分布($Y|X$)的拟合程度,例如由于模型设定错误、过拟合和高维性导致的问题。在本文中,我们考虑评估一般二元分类器拟合优度的基本问题。我们的框架不对条件分布 $Y|X$ 做任何参数假设,并将其视为只能通过查询访问的黑盒预言机模型。我们将拟合优度评估问题表述为如下形式的容差假设检验:\\[ H_0: \mathbb{E}\Big[D_f\Big({\sf Bern}(η(X))\\|{\sf Bern}(\hatη(X))\Big)\Big]\leq τ\\, \\],其中 $D_f$ 表示 $f$-散度函数,$η(x)$ 和 $hatη(x)$ 分别表示特征向量 $x$ 取正标签的真实似然和估计似然。我们提出了一种新颖的检验方法,称为 GRASP,用于检验 $H_0$,该方法适用于有限样本设置,无论特征如何(分布无关)。我们还提出了 model-X GRASP,专为模型X设置设计,其中特征向量的联合分布已知。Model-X GRASP 利用该分布信息以获得更好的功效。我们通过大量数值实验评估了我们的检验性能。

英文摘要:

Performance of classifiers is often measured in terms of average accuracy on test data. Despite being a standard measure, average accuracy fails in characterizing the fit of the model to the underlying conditional law of labels given the features vector ($Y|X$), e.g. due to model misspecification, over fitting, and high-dimensionality. In this paper, we consider the fundamental problem of assessing the goodness-of-fit for a general binary classifier. Our framework does not make any parametric assumption on the conditional law $Y|X$, and treats that as a black box oracle model which can be accessed only through queries. We formulate the goodness-of-fit assessment problem as a tolerance hypothesis testing of the form \[ H_0: \mathbb{E}\Big[D_f\Big({\sf Bern}(η(X))\|{\sf Bern}(\hatη(X))\Big)\Big]\leq τ\,, \] where $D_f$ represents an $f$-divergence function, and $η(x)$, $\hatη(x)$ respectively denote the true and an estimate likelihood for a feature vector $x$ admitting a positive label. We propose a novel test, called \grasp for testing $H_0$, which works in finite sample settings, no matter the features (distribution-free). We also propose model-X \grasp designed for model-X settings where the joint distribution of the features vector is known. Model-X \grasp uses this distributional information to achieve better power. We evaluate the performance of our tests through extensive numerical experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑