重新审视大型语言模型的不确定性估计与校准
Revisiting Uncertainty Estimation and Calibration of Large Language Models
AI总结：
本研究全面评估了80个大型语言模型在MMLU-Pro基准上的不确定性估计与校准，发现语言语言不确定性（LVU）优于其他方法，且高准确率不等于可靠的不确定性，强调需多视角评估以提升模型可靠性。
摘要：
随着大型语言模型（LLMs）日益广泛地部署于高风险应用中，稳健的不确定性估计对于确保LLMs安全且值得信赖的部署至关重要。我们开展了迄今为止最全面的LLMs不确定性估计研究，评估了80个模型，涵盖开源与闭源系列、稠密与混合专家（MoE）架构、推理与非推理模式、量化变体以及从0.6B到671B的参数规模。聚焦于三种代表性的黑盒单次方法，包括基于词元概率的不确定性（TPU）、数值语言不确定性（NVU）和语言语言不确定性（LVU），我们使用具有挑战性的MMLU-Pro基准系统地评估了不确定性校准和选择性分类，该基准涵盖推理密集型和基于知识的任务。结果表明，LVU始终优于TPU和NVU，提供更强的校准和判别力，同时具有更好的可解释性。我们还发现，高准确率并不意味着可靠的不确定性，且模型规模、后训练、推理能力和量化均会影响估计性能。值得注意的是，LLMs在推理任务上比在知识密集型任务上表现出更好的不确定性估计，而良好的校准并不必然转化为有效的错误排序。这些发现凸显了多视角评估的必要性，并将LVU定位为提升现实场景中LLMs可靠性的实用工具。
英文摘要：
As large language models (LLMs) are increasingly deployed in high-stakes applications, robust uncertainty estimation is essential for ensuring the safe and trustworthy deployment of LLMs. We present the most comprehensive study to date of uncertainty estimation in LLMs, evaluating 80 models spanning open- and closed-source families, dense and Mixture-of-Experts (MoE) architectures, reasoning and non-reasoning modes, quantization variants and parameter scales from 0.6B to 671B. Focusing on three representative black-box single-pass methods, including token probability-based uncertainty (TPU), numerical verbal uncertainty (NVU), and linguistic verbal uncertainty (LVU), we systematically evaluate uncertainty calibration and selective classification using the challenging MMLU-Pro benchmark, which covers both reasoning-intensive and knowledge-based tasks. Our results show that LVU consistently outperforms TPU and NVU, offering stronger calibration and discrimination while being more interpretable. We also find that high accuracy does not imply reliable uncertainty, and that model scale, post-training, reasoning ability and quantization all influence estimation performance. Notably, LLMs exhibit better uncertainty estimates on reasoning tasks than on knowledge-heavy ones, and good calibration does not necessarily translate to effective error ranking. These findings highlight the need for multi-perspective evaluation and position LVU as a practical tool for improving the reliability of LLMs in real-world settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
