arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.02742cs.LG
收藏

基于学习的有害协变量偏移假设检验

A Learning Based Hypothesis Test for Harmful Covariate Shift

University of Toronto(多伦多大学)

Tom Ginsberg, Zhongyuan Liang, Rahul G. Krishnan

摘要:

在测试时快速准确地识别协变量偏移的能力是部署在高风险领域中的安全机器学习系统的关键且经常被忽视的组成部分。虽然存在用于检测何时不应对分布外测试示例进行预测的方法,但识别训练和测试时间之间的分布水平差异可以帮助确定何时应从部署设置中删除模型并重新训练。在这项工作中,我们将有害的协变量偏移(HCS)定义为可能削弱预测模型泛化的分布变化。为了检测HCS,我们使用在训练数据上一致而在测试数据上不一致的分类器集合之间的不一致性。我们推导出一个损失函数训练这个合奏,并表明,不一致率和熵代表HCS强大的判别统计。从经验上讲,我们证明了我们的方法检测有害的协变量偏移与统计确定性的各种高维数据集的能力。在众多领域和模式,我们显示了最先进的性能相比,现有的方法,特别是当观察到的测试样本的数量是小的。

英文摘要:

The ability to quickly and accurately identify covariate shift at test time is a critical and often overlooked component of safe machine learning systems deployed in high-risk domains. While methods exist for detecting when predictions should not be made on out-of-distribution test examples, identifying distributional level differences between training and test time can help determine when a model should be removed from the deployment setting and retrained. In this work, we define harmful covariate shift (HCS) as a change in distribution that may weaken the generalization of a predictive model. To detect HCS, we use the discordance between an ensemble of classifiers trained to agree on training data and disagree on test data. We derive a loss function for training this ensemble and show that the disagreement rate and entropy represent powerful discriminative statistics for HCS. Empirically, we demonstrate the ability of our method to detect harmful covariate shift with statistical certainty on a variety of high-dimensional datasets. Across numerous domains and modalities, we show state-of-the-art performance compared to existing methods, particularly when the number of observed test samples is small.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑