Despite their incredible performance, it is well reported that deep neural networks tend to be overoptimistic about their prediction confidence. Finding effective and efficient calibration methods for neural networks is therefore an important endeavour towards better uncertainty quantification in deep learning. In this manuscript, we introduce a novel calibration technique named expectation consistency (EC), consisting of a post-training rescaling of the last layer weights by enforcing that the average validation confidence coincides with the average proportion of correct labels. First, we show that the EC method achieves similar calibration performance to temperature scaling (TS) across different neural network architectures and data sets, all while requiring similar validation samples and computational resources. However, we argue that EC provides a principled method grounded on a Bayesian optimality principle known as the Nishimori identity. Next, we provide an asymptotic characterization of both TS and EC in a synthetic setting and show that their performance crucially depends on the target function. In particular, we discuss examples where EC significantly outperforms TS.
A Finer Calibration Analysis for Adversarial Robustness
Google Research(谷歌研究院);Rutgers University(罗格斯大学);Courant Institute of Mathematical Sciences(库朗数学科学研究所)
我们提出了一个更一般的分析$H$校准对抗强大的分类。通过采用更精细的校准定义,我们可以覆盖在以前的工作中研究的限制性假设集之外的设置。特别是,我们的结果适用于机器学习中使用的最常见的假设集。我们都修正了一些先前的校准结果(Bao等人,2020)和推广其他(Awasthi et al.,2021年)。此外,我们的校准结果结合Awasthi等人(2021)先前的一致性研究,也导致了更一般的$H$一致性结果,涵盖了常见的假设集。
TCUQ: Single-Pass Uncertainty Quantification from Temporal Consistency with Streaming Conformal Calibration for TinyML
Multidisciplinary Faculty of Nador University Mohammed Premier Oujda(穆罕默德·布雷德·奥杰达大学多学科学院);Laboratory LaSTI, ENSAK Sultan Moulay Slimane University Khouribga(库尔希格大学拉斯特伊实验室)
Northwestern University(西北大学);Northeastern University(东北大学);Khoury College of Computer Sciences(科里学院计算机科学学院);Harvard University(哈佛大学);Microsoft Research, New England(微软研究院,新英格兰)
Consistency-Driven Calibration and Matching for Few-Shot Class-Incremental Learning
School of Automation, Central South University, China(中南大学自动化学院);Big Data Institute, Central South University, China(中南大学大数据研究院);School of Computer Science, Faculty of Engineering, The University of Sydney, Australia(悉尼大学工程学院计算机科学系)
Think Consistently, Reason Efficiently: Energy-Based Calibration for Implicit Chain-of-Thought
University of Oxford(牛津大学);Tsinghua University(清华大学);Tencent(腾讯);Southern University of Science and Technology(南方科技大学);National University of Singapore(新加坡国立大学)
Graph-based Confidence Calibration for Large Language Models
Department of Computer Science Tufts University(塔夫茨大学计算机科学系);Department of Computer Science Virginia Tech(弗吉尼亚理工大学计算机科学系);Department of Computer Science Virginia Tech, University of California, Davis(弗吉尼亚理工大学计算机科学系,加州大学戴维斯分校计算机科学系)