arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13540cs.CLcs.AIcs.LG
收藏

对大型语言模型的校准:从响应到能力

On Calibration of Large Language Models: From Response To Capability

Appier AI Research(Appier人工智能研究院);National Taiwan University(国立台湾大学)

Sin-Han Yang, Cheng-Kuang Wu, Chieh-Yen Lin, Yun-Nung Chen, Hung-yi Lee, Shao-Hua Sun

AI总结:

本文提出能力校准方法,用于改进大型语言模型在查询上的准确性估计,以提升预测和资源分配效果。

摘要:

大型语言模型(LLMs)被广泛部署为通用问题求解器,因此准确的置信度估计对于可靠使用至关重要。先前关于LLM校准的工作主要集中在响应层面的置信度,即估计单个生成输出的正确性。然而,这种形式与许多实际场景不匹配,其中核心问题是模型解决查询的整体可能性。我们证明,这种不匹配源于现代LLM解码的随机性,使得单个响应的正确性无法反映模型的能力。为了解决这个问题,我们引入了能力校准,旨在针对模型在查询上的预期准确性。我们正式区分了能力校准与响应校准,并展示了两者在理论和经验上都存在差异。我们建立了一个经验评估框架,并研究了一系列置信度估计方法。我们的结果表明,能力校准的置信度提高了pass@$k$预测和推理预算分配,为多样应用奠定了基础。

英文摘要:

Accurate confidence estimation is critical for reliable use of large language models (LLMs). Prior work on LLM calibration largely focuses on response-level confidence, which estimates the correctness of a single generated output. However, this formulation is misaligned with many practical settings where the central question is how likely a model is to solve a query overall. We show that this mismatch results from the stochastic nature of modern LLM decoding, under which single-response correctness fails to reflect underlying model capability. To address this issue, we introduce capability calibration, a new evaluation framework for measuring how well query-level confidence aligns with a model's expected accuracy on individual queries. We formally distinguish capability calibration (CC) from response calibration (RC) and show that the two differ both theoretically and empirically. We further show that CC is better suited than RC to applications like pass@k prediction and inference budget allocation. Finally, we evaluate common confidence estimation methods to understand the practical feasibility of CC.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑