Large language models (LLM) can state their confidence in an answer (verbalized confidence), or the confidence can be read from the probability of the answer's tokens (token confidence). Comparisons of the two ask which is better calibrated, but the token probability of an answer depends on the text that precedes it. Across five open LLMs and four question-answering datasets, we fix each answer, its verbalized confidence, and its correctness label, and vary how each score is measured. The scores themselves change with the protocol, and so does the comparison between them. Changing the context in which the token probability is read shifts the gap between the two scores by about the size of the gap itself (.287 vs. .304 in expected calibration error, ECE; .189 vs. .145 in AUROC). The 95% interval of the shift excludes zero in 19 and 17 of the 20 settings, and under AUROC the shift has the same sign in all of them, so the context changes how answers are ranked. The better score flips in 8 settings under each metric, counting only flips that the intervals support, and the shift also appears in four larger models of up to 72B parameters. Most of the shift appears with changes to the instruction, not to the position of the answer. Other measurement choices also matter, and stated confidence depends on how a supplied answer is written. We turn these results into a reporting checklist for LLM confidence and its comparisons.
Fact-Level Confidence Calibration and Self-Correction
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Penn State University(宾夕法尼亚州立大学)
Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation
University of Science and Technology of China(中国科学技术大学);SKL of Processors, Institute of Computing Technology, CAS(中国科学院计算技术研究所处理器专项实验室);National University of Singapore(新加坡国立大学);University of Chinese Academy of Sciences(中国科学院大学);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQA
College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);Zhihui Medical Technology (Shanghai) Co., Ltd.(智汇医疗科技(上海)有限公司)
Double-Calibration: Towards Reliable LLMs via Calibrating Knowledge and Reasoning Confidence
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China(中山大学计算机科学与工程学院,广州,中国);Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR(香港理工大学计算机系,香港特别行政区);School of Science and Technology, Hong Kong Metropolitan University, Hong Kong SAR(香港 Metropolitan 大学科技学院,香港特别行政区)
Graph-based Confidence Calibration for Large Language Models
Department of Computer Science Tufts University(塔夫茨大学计算机科学系);Department of Computer Science Virginia Tech(弗吉尼亚理工大学计算机科学系);Department of Computer Science Virginia Tech, University of California, Davis(弗吉尼亚理工大学计算机科学系,加州大学戴维斯分校计算机科学系)
Dynamic Alignment and Calibration for Multimodal Learning
School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);Tianyijiaotong Technology Ltd.(天翼交通科技有限公司);School of Computer Science and Technology, Hainan University(海南大学计算机科学与技术学院)
The Confidence Trap: Calibration Attacks for Graph Neural Networks
Virginia Polytechnic Institute and State University(弗吉尼亚理工学院暨州立大学);The Pennsylvania State University(宾夕法尼亚州立大学);VinUniversity;University of Illinois at Chicago(伊利诺伊大学芝加哥分校)
Calibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inference
King’s Communications, Learning & Information Processing (KCLIP) lab(国王通讯、学习与信息处理(KCLIP)实验室);Centre for Intelligent Information Processing Systems (CIIPS)(智能信息处理系统中心);Department of Engineering, King’s College London(伦敦国王学院工程系)
Probing Routing-Conditional Calibration in Attention-Residual Transformers
Adelaide University(阿德莱德大学);Australian Institute for Machine Learning (AIML), Adelaide University(澳大利亚机器学习研究所(AIML),阿德莱德大学);The University of Queensland(昆士兰大学)