arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2012.12670math.STstat.TH
收藏

测试学习过程是否经过校准

Testing whether a Learning Procedure is Calibrated

Jon Cockayne, Matthew M. Graham, Chris J. Oates, T. J. Sullivan, Onur Teymur

摘要:

学习过程将数据集作为输入,并对被假设为产生数据集的模型的参数$θ$执行推断。在这里,我们考虑学习过程,其输出是一个概率分布,表示在看到数据集后关于$θ$的不确定性。贝叶斯推理是这种过程的一个主要例子,但也可以构建其他返回分布输出的学习过程。本文研究了学习过程被视为已校准的条件,从这个意义上说,真正的数据生成参数作为其分布输出的样本是合理的。如果一个学习过程的推理和预测系统性地过度自信或自信不足,那么它就无法被校准。另一方面,被校准的学习过程不需要是统计上有效的。一个假设测试框架的开发,以评估,使用模拟,是否校准的学习过程。本报告介绍了几个小插图,以说明该框架的不同方面。

英文摘要:

A learning procedure takes as input a dataset and performs inference for the parameters $θ$ of a model that is assumed to have given rise to the dataset. Here we consider learning procedures whose output is a probability distribution, representing uncertainty about $θ$ after seeing the dataset. Bayesian inference is a prime example of such a procedure, but one can also construct other learning procedures that return distributional output. This paper studies conditions for a learning procedure to be considered calibrated, in the sense that the true data-generating parameters are plausible as samples from its distributional output. A learning procedure whose inferences and predictions are systematically over- or under-confident will fail to be calibrated. On the other hand, a learning procedure that is calibrated need not be statistically efficient. A hypothesis-testing framework is developed in order to assess, using simulation, whether a learning procedure is calibrated. Several vignettes are presented to illustrate different aspects of the framework.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑