超越“我不知道”：评估LLM在区分数据和模型不确定性中的自我意识
Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
AI总结：
本文提出UA-Bench基准测试，评估LLM在区分数据不确定性和模型不确定性方面的能力，发现即使先进模型也难以可靠区分，并提出轻量数据合成与强化学习策略提升不确定性判断。
摘要：
可靠的大型语言模型（LLMs）应在信心不足时回避回答。然而，先前研究往往将拒绝视为通用的“我不知道”，未能区分输入层面的模糊性（数据不确定性）与能力限制（模型不确定性）。这种区分不足限制了后续行动决策，如请求澄清或调用外部工具。本文引入UA-Bench，一个包含超过3500个问题的基准测试，涵盖知识密集型和推理密集型任务，旨在评估显式不确定性归因。对18种前沿LLM的评估显示，即使最先进的模型也难以可靠区分数据不确定性和模型不确定性，且高回答准确性并不必然意味着强不确定性归因能力。为缩小这一差距，我们提出了一种轻量级数据合成与强化学习策略。在Qwen3-4B-Instruct-2507和Qwen3-8B的思考模式下进行的实验表明，所提出的方法在提升不确定性归因能力的同时保持了回答准确性。我们的代码和数据现已公开。
英文摘要：
Reliable Large Language Models (LLMs) should abstain when confidence is insufficient. However, prior studies often treat refusal as a generic "I don't know'', failing to distinguish input-level ambiguity (data uncertainty) from capability limitations (model uncertainty). This lack of distinction limits downstream action decisions like requesting clarification or invoking external tools. In this work, we introduce UA-Bench, a benchmark of over 3,500 questions drawn from six datasets spanning knowledge-intensive and reasoning-intensive tasks, designed to evaluate explicit uncertainty attribution. An evaluation of 18 frontier LLMs shows that even state-of-the-art models struggle to reliably discriminate between data uncertainty and model uncertainty, and that high answer accuracy does not necessarily imply strong uncertainty attribution ability. To narrow this gap, we propose a lightweight data synthesis and reinforcement learning strategy. Experiments on both Qwen3-4B-Instruct-2507 and Qwen3-8B in thinking mode show that the proposed method improves uncertainty attribution while preserving answer accuracy. Our code and data are publicly available now.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
