arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2206.10982stat.MLcs.LG
收藏

样本外模型评估的诊断工具

Diagnostic Tool for Out-of-Sample Model Evaluation

Uppsala University(乌普萨拉大学)

Ludvig Hult, Dave Zachariah, Petre Stoica

AI总结:

本文提出一种基于有限校准数据的模型诊断工具,在弱假设下提供有限样本保证,可量化分布偏移影响、辅助回归分析并支持模型选择与超参数调优。

摘要:

模型拟合度的评估是机器学习的关键部分。标准范式是通过在训练数据上平均最小化所选损失函数来学习模型,旨在未来数据上实现较小的损失。在本文中,我们考虑使用有限的校准数据集来刻画模型的未来样本外损失。我们提出了一种简单的模型诊断工具,在弱假设下提供有限样本保证。该工具易于计算和解释。我们进行了若干数值实验,以展示所提方法如何量化分布偏移的影响、辅助回归分析,并实现模型选择以及超参数调优。

英文摘要:

Assessment of model fitness is a key part of machine learning. The standard paradigm is to learn models by minimizing a chosen loss function averaged over training data, with the aim of achieving small losses on future data. In this paper, we consider the use of a finite calibration data set to characterize the future, out-of-sample losses of a model. We propose a simple model diagnostic tool that provides finite-sample guarantees under weak assumptions. The tool is simple to compute and to interpret. Several numerical experiments are presented to show how the proposed method quantifies the impact of distribution shifts, aids the analysis of regression, and enables model selection as well as hyper-parameter tuning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑