arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.06604cs.LGcs.CL
收藏

知何时错误:将置信度与正确性对齐以用于LLM错误检测

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

Alexa AI, Amazon, Seattle, WA, USA(Alexa AI,亚马逊,西雅图,华盛顿州)

Xie Xiaohu, Liu Xiaohu, Yao Benjamin

AI总结:

本文提出归一化置信度分数和自我评估框架,通过SFT提升LLM置信度可靠性,减少校准误差,提升错误检测效果。

摘要:

随着大型语言模型(LLMs)越来越多地应用于关键决策系统,缺乏可靠方法来衡量其不确定性是一个根本的信任风险。我们介绍了一个基于输出锚定标记概率的归一化置信度分数:对于结构化任务使用分类标签,对于开放式生成任务使用自我评估响应(是/否)。这使能够通过最小的开销和不依赖外部验证直接检测错误和幻觉。我们做出了三个关键贡献。首先,我们提出了一个归一化置信度分数和自我评估框架,该框架在七个多样化的基准任务和五个架构和规模各异的LLM上提供了可靠的置信度估计用于错误检测。第二,我们的理论分析揭示了监督微调(SFT)通过最大似然估计产生良好的校准置信度,而强化学习方法(PPO、GRPO)和DPO通过奖励利用产生过度自信。第三,我们提出了一种后RL SFT与自我蒸馏,以恢复RL训练模型的置信度可靠性。实验结果表明,SFT将平均置信度-正确性AUROC从0.806提高到0.879,并将校准误差从0.163降低到0.034在Qwen3-4B上,而GRPO和DPO降低了置信度可靠性。我们通过自适应检索增强生成(RAG)展示了实际价值,当模型缺乏置信度时,它只使用58%的检索操作就可恢复TriviaQA上最大可实现的准确率提升的95%。

英文摘要:

As large language models (LLMs) are increasingly deployed in critical decision-making systems, the lack of reliable methods to measure their uncertainty presents a fundamental trustworthiness risk. We introduce a normalized confidence score based on output anchor token probabilities: classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This enables direct detection of errors and hallucinations with minimal overhead and without external validation. We make three key contributions. First, we propose a normalized confidence score and self-evaluation framework that exposes reliable confidence estimates for error detection across seven diverse benchmark tasks and five LLMs of varying architectures and sizes. Second, our theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence through maximum-likelihood estimation, whereas reinforcement learning methods (PPO, GRPO) and DPO induce overconfidence via reward exploitation. Third, we propose post-RL SFT with self-distillation to restore confidence reliability in RL-trained models. Empirical results demonstrated that SFT improved average confidence-correctness AUROC from 0.806 to 0.879 and reduced calibration error from 0.163 to 0.034 on Qwen3-4B, while GRPO and DPO degraded confidence reliability. We demonstrated practical value through adaptive retrieval-augmented generation (RAG) that selectively retrieves context when the model lacks confidence, using only 58\% of retrieval operations to recover 95\% of the maximum achievable accuracy gain on TriviaQA

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑