arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.02644cs.LGstat.ML
收藏

神经网络校准的期望一致性

Expectation consistency for calibration of neural networks

École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院);École Normale Supérieure - PSL(巴黎高等师范学院-PSL大学);CNRS(法国国家科学研究中心)

Lucas Clarté, Bruno Loureiro, Florent Krzakala, Lenka Zdeborová

AI总结:

本文提出期望一致性(EC)校准方法,通过重缩放末层权重使验证置信度与正确率一致,性能与温度缩放相当,但基于Nishimori恒等式更具原则性,并在某些目标函数下显著优于温度缩放。

摘要:

尽管深度神经网络性能卓越,但已有充分研究表明其预测置信度往往过于乐观。因此,寻找有效且高效的神经网络校准方法,对于改善深度学习中的不确定性量化具有重要意义。本文提出了一种新颖的校准技术,称为期望一致性(EC),该方法通过强制验证集平均置信度与正确标签的平均比例一致,对最后一层权重进行训练后重缩放。首先,我们证明在不同神经网络架构和数据集上,EC方法能达到与温度缩放(TS)相当的校准性能,同时所需的验证样本和计算资源也相近。然而,我们认为EC提供了一种基于贝叶斯最优性原则(即Nishimori恒等式)的 principled 方法。接下来,我们在合成环境中对TS和EC进行了渐近特征刻画,并表明它们的性能关键取决于目标函数。特别是,我们讨论了EC显著优于TS的实例。

英文摘要:

Despite their incredible performance, it is well reported that deep neural networks tend to be overoptimistic about their prediction confidence. Finding effective and efficient calibration methods for neural networks is therefore an important endeavour towards better uncertainty quantification in deep learning. In this manuscript, we introduce a novel calibration technique named expectation consistency (EC), consisting of a post-training rescaling of the last layer weights by enforcing that the average validation confidence coincides with the average proportion of correct labels. First, we show that the EC method achieves similar calibration performance to temperature scaling (TS) across different neural network architectures and data sets, all while requiring similar validation samples and computational resources. However, we argue that EC provides a principled method grounded on a Bayesian optimality principle known as the Nishimori identity. Next, we provide an asymptotic characterization of both TS and EC in a synthetic setting and show that their performance crucially depends on the target function. In particular, we discuss examples where EC significantly outperforms TS.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑