Predicting calibrated confidence scores for multi-class deep networks is important for avoiding rare but costly mistakes. A common approach is to learn a post-hoc calibration function that transforms the output of the original network into calibrated confidence scores while maintaining the network's accuracy. However, previous post-hoc calibration techniques work only with simple calibration functions, potentially lacking sufficient representation to calibrate the complex function landscape of deep networks. In this work, we aim to learn general post-hoc calibration functions that can preserve the top-k predictions of any deep network. We call this family of functions intra order-preserving functions. We propose a new neural network architecture that represents a class of intra order-preserving functions by combining common neural network components. Additionally, we introduce order-invariant and diagonal sub-families, which can act as regularization for better generalization when the training data size is small. We show the effectiveness of the proposed method across a wide range of datasets and classifiers. Our method outperforms state-of-the-art post-hoc calibration methods, namely temperature scaling and Dirichlet calibration, in several evaluation metrics for the task.
Texas A&M University（德克萨斯农工大学）；Adobe Research（奥多比研究院）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）
Generative adversarial networks (GANs) nowadays are capable of producing images of incredible realism. One concern raised is whether the state-of-the-art GAN's learned distribution still suffers from mode collapse, and what to do if so. Existing diversity tests of samples from GANs are usually conducted qualitatively on a small scale, and/or depends on the access to original training data as well as the trained model parameters. This paper explores to diagnose GAN intra-mode collapse and calibrate that, in a novel black-box setting: no access to training data, nor the trained model parameters,…
Deep intra-operative illumination calibration of hyperspectral cameras
Siemens AG（西门子股份公司）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Helmholtz Information and Data Science School for Health（亥姆霍兹健康信息与数据科学学院）；Heidelberg University（海德堡大学）；National Center for Tumor Diseases (NCT)（国家肿瘤疾病中心）；Heidelberg University Hospital（海德堡大学医院）；University Medical Center Mannheim（曼海姆大学医学中心）；DKFZ Hector Cancer Institute at the University Medical Center Mannheim（德国癌症研究中心赫克曼海姆大学医学中心癌症研究所）
Technical University of Munich（慕尼黑工业大学）；German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；Goethe University Frankfurt（法兰克福歌德大学）
We address the problem of uncertainty calibration and introduce a novel calibration method, Parametrized Temperature Scaling (PTS). Standard deep neural networks typically yield uncalibrated predictions, which can be transformed into calibrated confidence scores using post-hoc calibration methods. In this contribution, we demonstrate that the performance of accuracy-preserving state-of-the-art post-hoc calibrators is limited by their intrinsic expressive power. We generalize temperature scaling by computing prediction-specific temperatures, parameterized by a neural network. We show with exten…
Rethinking Post-Hoc Calibration in Semantic Segmentation
ICube Laboratory, CNRS UMR 7357, University of Strasbourg（斯特拉斯堡大学ICube实验室，CNRS UMR 7357）；CLCC Institut Strauss（斯特劳斯研究所）；German Cancer Research Center (DKFZ)（德国癌症研究中心）；Faculty of Mathematics and Computer Science, University of Heidelberg（海德堡大学数学与计算机科学学院）；Medical Faculty Heidelberg, Heidelberg University（海德堡大学医学院）；Pattern Analysis and Learning Group, Dept. of Radiation Oncology, Heidelberg University Hospital（海德堡大学医院放射肿瘤科模式分析与学习组）
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；IBM Research（IBM研究院）
Bayesian decision theory provides an elegant framework for acting optimally under uncertainty when tractable posterior distributions are available. Modern Bayesian models, however, typically involve intractable posteriors that are approximated with, potentially crude, surrogates. This difficulty has engendered loss-calibrated techniques that aim to learn posterior approximations that favor high-utility decisions. In this paper, focusing on Bayesian neural networks, we develop methods for correcting approximate posterior predictive distributions encouraging them to prefer high-utility decisions…
Beyond Appearance Shifts: Task-Semantic Action Calibration for VLA Models
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；National Centre for Computer Animation, Bournemouth University（伯恩茅斯大学国家计算机动画中心）；Shanghai Jiao Tong University（上海交通大学）；Eastern Institute of Technology, Ningbo（宁波东方理工大学）
Lawrence Livermore National Laboratories（劳伦斯利弗莫尔国家实验室）
This paper studies the problem of post-hoc calibration of machine learning classifiers. We introduce the following desiderata for uncertainty calibration: (a) accuracy-preserving, (b) data-efficient, and (c) high expressive power. We show that none of the existing methods satisfy all three requirements, and demonstrate how Mix-n-Match calibration strategies (i.e., ensemble and composition) can help achieve remarkably better data-efficiency and expressive power while provably maintaining the classification accuracy of the original classifier. Mix-n-Match strategies are generic in the sense that…
Bosch Center for Artificial Intelligence（博世人工智能中心）；University of Stuttgart（斯图加特大学）
Post-hoc multi-class calibration is a common approach for providing high-quality confidence estimates of deep neural network predictions. Recent work has shown that widely used scaling methods underestimate their calibration error, while alternative Histogram Binning (HB) methods often fail to preserve classification accuracy. When classes have small prior probabilities, HB also faces the issue of severe sample-inefficiency after the conversion into K one-vs-rest class-wise calibration problems. The goal of this paper is to resolve the identified issues of HB in order to provide calibrated con…
While deep neural networks provide good performance for a range of challenging tasks, calibration and uncertainty estimation remain major challenges, especially under distribution shift. In this paper, we propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation, calibration, and out-of-distribution robustness with deep networks. Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principl…
Beijing University of Posts and Telecommunications（北京邮电大学）
Despite Graph Neural Networks (GNNs) have achieved remarkable accuracy, whether the results are trustworthy is still unexplored. Previous studies suggest that many modern neural networks are over-confident on the predictions, however, surprisingly, we discover that GNNs are primarily in the opposite direction, i.e., GNNs are under-confident. Therefore, the confidence calibration for GNNs is highly desired. In this paper, we propose a novel trustworthy GNN model by designing a topology-aware post-hoc calibration function. Specifically, we first verify that the confidence distribution in a graph…
University at Buffalo（布法罗大学）；Penn State Erie（宾夕法尼亚州立大学伊利分校）
Deep Neural Networks (DNNs), despite their tremendous success in recent years, could still cast doubts on their predictions due to the intrinsic uncertainty associated with their learning process. Ensemble techniques and post-hoc calibrations are two types of approaches that have individually shown promise in improving the uncertainty calibration of DNNs. However, the synergistic effect of the two types of methods has not been well explored. In this paper, we propose a truth discovery framework to integrate ensemble-based and post-hoc calibration methods. Using the geometric variance of the en…
We propose a restricted class of tensor network state, built from number-state preserving tensors, for supervised learning tasks. This class of tensor network is argued to be a natural choice for classifiers as (i) they map classical data to classical data, and thus preserve the interpretability of data under tensor transformations, (ii) they can be efficiently trained to maximize their scalar product against classical data sets, and (iii) they seem to be as powerful as generic (unrestricted) tensor networks in this task. Our proposal is demonstrated using a variety of benchmark classification…
École Normale Supérieure（巴黎高等师范学院）；Chalmers University of Technology（查尔姆斯理工大学）；National University of Singapore（新加坡国立大学）
The calibration of noise for a privacy-preserving mechanism depends on the sensitivity of the query and the prescribed privacy level. A data steward must make the non-trivial choice of a privacy level that balances the requirements of users and the monetary constraints of the business entity. We analyse roles of the sources of randomness, namely the explicit randomness induced by the noise distribution and the implicit randomness induced by the data-generation distribution, that are involved in the design of a privacy-preserving mechanism. The finer analysis enables us to provide stronger priv…
Structure-Preserving Digital Twins via Conditional Neural Whitney Forms
Applied Mechanics, University of Pennsylvania, Philadelphia, PA, USA（应用力学，宾夕法尼亚大学，费城，PA，美国）；Sandia National Laboratories, Albuquerque, NM, USA（桑迪亚国家实验室，阿尔伯克基，NM，美国）
The Confidence Trap: Calibration Attacks for Graph Neural Networks
Virginia Polytechnic Institute and State University（弗吉尼亚理工学院暨州立大学）；The Pennsylvania State University（宾夕法尼亚州立大学）；VinUniversity；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）
Network alignment, the process of finding correspondences between nodes in different graphs, has many scientific and industrial applications. Existing unsupervised network alignment methods find suboptimal alignments that break up node neighborhoods, i.e. do not preserve matched neighborhood consistency. To improve this, we propose CONE-Align, which models intra-network proximity with node embeddings and uses them to match nodes across networks after aligning the embedding subspaces. Experiments on diverse, challenging datasets show that CONE-Align is robust and obtains 19.25% greater accuracy…
The Hong Kong University of Science and Technology（香港科技大学）
The conventional design of wireless communication systems typically relies on established mathematical models that capture the characteristics of different communication modules. Unfortunately, such design cannot be easily and directly applied to future wireless networks, which will be characterized by large-scale ultra-dense networks whose design complexity scales exponentially with the network size. Furthermore, such networks will vary dynamically in a significant way, which makes it intractable to develop comprehensive analytical models. Recently, deep learning-based approaches have emerged…
Indiana University（印第安纳大学）；Electronics and Telecommunications Research Institute（韩国电子通信研究院）
Conventional audio coding technologies commonly leverage human perception of sound, or psychoacoustics, to reduce the bitrate while preserving the perceptual quality of the decoded audio signals. For neural audio codecs, however, the objective nature of the loss function usually leads to suboptimal sound quality as well as high run-time complexity due to the large model size. In this work, we present a psychoacoustic calibration scheme to re-define the loss functions of neural audio coding systems so that it can decode signals more perceptually similar to the reference, yet with a much lower m…
