Deep learning has achieved the state-of-the-art performance across medical imaging tasks; however, model calibration is often not considered. Uncalibrated models are potentially dangerous in high-risk applications since the user does not know when they will fail. Therefore, this paper proposes a novel domain-aware loss function to calibrate deep learning models. The proposed loss function applies a class-wise penalty based on the similarity between classes within a given target domain. Thus, the approach improves the calibration while also ensuring that the model makes less risky errors even when incorrect. The code for this software is available at https://github.com/lab-smile/DOMINO.
University of Florida（佛罗里达大学）；Herbert Wertheim College of Engineering（赫伯特·韦特海姆工程学院）；McKnight Brain Institute（麦克奈特脑研究所）；College of Public Health and Health Professions（公共卫生与健康职业学院）；College of Medicine（医学院）；United States Air Force Research Laboratory（美国空军研究实验室）
Model calibration measures the agreement between the predicted probability estimates and the true correctness likelihood. Proper model calibration is vital for high-risk applications. Unfortunately, modern deep neural networks are poorly calibrated, compromising trustworthiness and reliability. Medical image segmentation particularly suffers from this due to the natural uncertainty of tissue boundaries. This is exasperated by their loss functions, which favor overconfidence in the majority classes. We address these challenges with DOMINO, a domain-aware model calibration method that leverages …
DOMINO++: Domain-aware Loss Regularization for Deep Learning Generalizability
University of Florida (UF)（佛罗里达大学）；Torch Technologies, LLC（Torch科技有限责任公司）；McKnight Brain Institute（麦克奈特脑研究所）；College of Public Health and Health Professions（公共卫生与健康职业学院）；College of Medicine（医学院）；United States Air Force Research Laboratory（美国空军研究实验室）；Herbert Wertheim College of Engineering（赫伯特·韦特海姆工程学院）；Department of Biomedical Engineering（生物医学工程系）；Department of Clinical and Health Psychology（临床与健康心理学系）；Department of Neuroscience（神经科学系）；Department of Electrical and Computer Engineering（电气与计算机工程系）；Department of Computer Information and Science and Engineering（计算机信息与工程系）
Machine learning models that achieve high overall accuracy often make systematic errors on important subsets (or slices) of data. Identifying underperforming slices is particularly challenging when working with high-dimensional inputs (e.g. images, audio), where important slices are often unlabeled. In order to address this issue, recent studies have proposed automated slice discovery methods (SDMs), which leverage learned model representations to mine input data for slices on which a model performs poorly. To be useful to a practitioner, these methods must identify slices that are both underp…
As we move away from the data, the predictive uncertainty should increase, since a great variety of explanations are consistent with the little available information. We introduce Distance-Aware Prior (DAP) calibration, a method to correct overconfidence of Bayesian deep learning models outside of the training domain. We define DAPs as prior distributions over the model parameters that depend on the inputs through a measure of their distance from the training set. DAP calibration is agnostic to the posterior inference method, and it can be performed as a post-processing step. We demonstrate it…
Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigation
Johns Hopkins University, School of Medicine（约翰霍普金斯大学医学院）；Massachusetts Institute of Technology（麻省理工学院）；Microsoft Healthcare & Life Sciences（微软医疗健康与生命科学）
Optimizing Calibration by Gaining Aware of Prediction Correctness
The Australian National University（澳大利亚国立大学）；Data61, The Commonwealth Scientific and Industrial Research Organisation（联邦科学与工业研究组织Data61）；The Hong Kong Polytechnic University（香港理工大学）；Stanford University（斯坦福大学）
Uncertainty Aware Training to Improve Deep Learning Model Calibration for Classification of Cardiac MR Images
School of Biomedical Engineering & Imaging Sciences, King’s College London（伦敦国王学院生物医学工程与影像科学学院）；Guy's and St Thomas' Hospital（盖伊和圣托马斯医院）；BioMedIA Group, Department of Computing, Imperial College London（帝国理工学院计算系生物医学图像分析组）
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；IIIS, Tsinghua University（清华大学交叉信息研究院）；Tencent（腾讯）
It is often observed that the probabilistic predictions given by a machine learning model can disagree with averaged actual outcomes on specific subsets of data, which is also known as the issue of miscalibration. It is responsible for the unreliability of practical machine learning systems. For example, in online advertising, an ad can receive a click-through rate prediction of 0.1 over some population of users where its actual click rate is 0.15. In such cases, the probabilistic predictions have to be fixed before the system can be deployed. In this paper, we first introduce a new evaluation…
Bosch Center for AI（博世人工智能中心）；University of Tübingen（蒂宾根大学）；University of Mannheim（曼海姆大学）；Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Bosch Center for Artificial Intelligence（博世人工智能中心）
Task-Aware Calibration: Provably Optimal Decoding in LLMs
School of Computation, Information & Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术学院）；Munich Data Science Institute（慕尼黑数据科学研究所）；Munich Center for Machine Learning（慕尼黑机器学习中心）；University of Amsterdam（阿姆斯特丹大学）
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；Noah’s Ark Lab, Shenzhen, Huawei Technologies Co. Ltd（华为技术有限公司诺亚方舟实验室）
Real-world data usually present long-tailed distributions. Training on imbalanced data tends to render neural networks perform well on head classes while much worse on tail classes. The severe sparseness of training instances for the tail classes is the main challenge, which results in biased distribution estimation during training. Plenty of efforts have been devoted to ameliorating the challenge, including data re-sampling and synthesizing new training instances for tail classes. However, no prior research has exploited the transferable knowledge from head classes to tail classes for calibra…
Semantic-Aware Gaussian Process Calibration with Structured Layerwise Kernels for Deep Neural Networks
Next Generation Defence Multidisciplinary Technology Research Center, Pohang University of Science and Technology（下一代国防多学科技术研究中心，POSTECH大学）；Department of Electrical Engineering, Pohang University of Science and Technology（电气工程系，POSTECH大学）
University of Technology Sydney（悉尼科技大学）；School of Electrical and Data Engineering（电气与数据工程学院）
The Jaccard index, also known as Intersection-over-Union (IoU score), is one of the most critical evaluation metrics in medical image segmentation. However, directly optimizing the mean IoU (mIoU) score over multiple objective classes is an open problem. Although some algorithms have been proposed to optimize its surrogates, there is no guarantee provided for their generalization ability. In this paper, we present a novel data-distribution-aware margin calibration method for a better generalization of the mIoU over the whole data-distribution, underpinned by a rigid lower bound. This scheme en…
Uncertainty-Aware Online Extrinsic Calibration: A Conformal Prediction Approach
Université de technologie de Compiègne, CNRS, Heudiasyc（法国图卢兹工学院、CNRS、Heudiasyc）；CNRS, INSA Lyon, UCBL, LIRIS, UMR5205（CNRS、里昂国立应用科学学院、UCBL、LIRIS、UMR5205）
This paper presents a novel calibration algorithm for Multi-Focus Plenoptic Cameras (MFPCs) using raw images only. The design of such cameras is usually complex and relies on precise placement of optic elements. Several calibration procedures have been proposed to retrieve the camera parameters but relying on simplified models, reconstructed images to extract features, or multiple calibrations when several types of micro-lens are used. Considering blur information, we propose a new Blur Aware Plenoptic (BAP) feature. It is first exploited in a pre-calibration step that retrieves initial camera…
Imperial College London（伦敦帝国理工学院）；National University of Singapore（新加坡国立大学）
Despite impressive accuracy, deep neural networks are often miscalibrated and tend to overly confident predictions. Recent techniques like temperature scaling (TS) and label smoothing (LS) show effectiveness in obtaining a well-calibrated model by smoothing logits and hard labels with scalar factors, respectively. However, the use of uniform TS or LS factor may not be optimal for calibrating models trained on a long-tailed dataset where the model produces overly confident probabilities for high-frequency classes. In this study, we propose class-distribution-aware TS (CDA-TS) and LS (CDA-LS) by…
PA-TCNet: Pathology-Aware Temporal Calibration with Physiology-Guided Target Refinement for Cross-Subject Motor Imagery EEG Decoding in Stroke Patients
School of Artificial Intelligence, Chongqing University of Technology（重庆理工大学人工智能学院）；Chongqing Key Laboratory of Embodied Intelligence Perception and Autonomous Learning for Humanoid Robots（重庆 embodied 智能感知与人形机器人自主学习关键实验室）；Key Laboratory of Advanced Equipment Intelligence of the Chongqing Education Commission（重庆市教育委员会先进设备智能关键实验室）；School of Smart Health, Chongqing Polytechnic University of Electronic Technology（重庆电子工程职业大学智能健康学院）；Department of Language Science and Technology, The Hong Kong Polytechnic University（香港理工大学语言科学与技术系）；School of Pharmacy and Bioengineering, Chongqing University of Technology（重庆理工大学药学院与生物工程学院）；School of Computer Science and Engineering, Chongqing University of Technology（重庆理工大学计算机科学与工程学院）
Budget-Aware Uncertainty for Radiotherapy Segmentation QA Using nnU-Net
Department of Biomedical Sciences, Humanitas University（人类医学科学系，人类itas大学）；Radiotherapy and Radiosurgery Department, IRCCS Humanitas Research Hospital（放疗与放射外科部门，IRCCS人类itas研究医院）
