Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Faculty of Computer Science, UniVie Doctoral School Computer Science（计算机科学系，维也纳大学计算机科学博士学院）；Faculty of Philological and Cultural Studies, University of Vienna, Austria（文学与文化研究系，维也纳大学，奥地利）；ILLC, University of Amsterdam, Netherlands（阿姆斯特丹大学ILLC，荷兰）；Khoury College of Computer Sciences, Northeastern University, USA（东北大学计算机科学学院，美国）；LMU Munich, Munich Center for Machine Learning (MCML), Germany（慕尼黑大学，慕尼黑机器学习中心（MCML），德国）
Yuxi Xia, Dennis Ulmer, Terra Blevins, Yihong Liu, Hinrich Schütze, Benjamin Roth
Confidence estimation (CE) indicates how reliable the answers of large language models are and impacts user trust and decision-making. Existing evaluations mainly concern the alignment between confidence and correctness, but ignore the variability of language: confidence estimates should remain consistent under semantically equivalent prompts or answer variations, while changing when answer meaning differs, as this may indicate a change in correctness. Therefore, we introduce a novel evaluation framework based on three complementary properties: \textbf{robustness} to prompt perturbations, \textbf{stability} across semantically equivalent answers, and \textbf{sensitivity} to semantically different answers. We show that these metrics are largely independent from existing CE metrics, and that common CE methods often fail on them: while most methods achieve high robustness and stability, they struggle to distinguish semantically different answers, potentially because they do not effectively leverage generation-side information. Overall, our framework exposes overlooked limitations of current CE evaluations and provides guidance for selecting confidence estimators for real-world applications.
Shared Experience, Separate Learning: Companion Confidence Calibration for LLMs
State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）
N2C2: Nearest Neighbor Enhanced Confidence Calibration for Cross-Lingual In-Context Learning
School of Informatics, University of Edinburgh（爱丁堡大学信息学院）；School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）
Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
Tsinghua University（清华大学）；Vulcan Research（伏尔坎研究院）；AIFT；Keio Global Research Institute (KGRI)（庆应全球研究所（KGRI））；China Mobile Research Institute（中国移动研究院）；Zhongguancun Laboratory（中关村实验室）
Evaluating and Calibrating LLM Confidence on Questions with Multiple Correct Answers
State Key Laboratory of AI Safety（人工智能安全国家重点实验室）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tsinghua University（清华大学）；Renmin University of China（中国人民大学）
School of Computer Science, University of Sydney（悉尼大学计算机科学学院）；City University of Hong Kong（香港城市大学）；Shanghai Jiao Tong University（上海交通大学）；University of Oxford, Department of Engineering Science（牛津大学工程科学系）；Department of Engineering Science, University of Oxford（牛津大学工程科学系）
Fact-Level Confidence Calibration and Self-Correction
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Penn State University（宾夕法尼亚州立大学）
Double-Calibration: Towards Reliable LLMs via Calibrating Knowledge and Reasoning Confidence
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China（中山大学计算机科学与工程学院，广州，中国）；Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR（香港理工大学计算机系，香港特别行政区）；School of Science and Technology, Hong Kong Metropolitan University, Hong Kong SAR（香港 Metropolitan 大学科技学院，香港特别行政区）
Graph-based Confidence Calibration for Large Language Models
Department of Computer Science Tufts University（塔夫茨大学计算机科学系）；Department of Computer Science Virginia Tech（弗吉尼亚理工大学计算机科学系）；Department of Computer Science Virginia Tech, University of California, Davis（弗吉尼亚理工大学计算机科学系，加州大学戴维斯分校计算机科学系）
Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards
Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing, China（中国科学院软件研究所信息处理实验室）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学）；Institute of Information Engineering, Chinese Academy of Sciences, Beijing, China（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences, Beijing, China（中国科学院大学网络安全学院）；National Computer Network Emergency Response Technical Team/Coordination Center of China, Beijing, China（中国国家计算机网络应急技术配合中心）
