基于元认知反馈的强化学习引发LLM忠实的不确定性表达
Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs
AI总结：
提出强化学习与元认知反馈（RLMF）和元认知数据选择方法，通过模型自我判断质量优化偏好学习，实现忠实校准（FC），在保持准确性的同时显著提升LLM不确定性表达的忠实度。
摘要：
元认知是智能的关键组成部分，描述了监控和调节自身认知过程的能力。然而，LLM在关键的元认知能力上表现出系统性缺陷：它们以高置信度产生幻觉，未能识别知识边界，并错误表达内部不确定性——这削弱了可信度和可靠性。由于监控任务表现并相应调整行为是元认知的核心，我们假设能够准确判断自身表现的模型更有可能改进表现。我们通过两种新颖机制实现这一想法：基于元认知反馈的强化学习（RLMF），一种在偏好优化期间根据模型自我判断表现的质量细化完成排序的范式；以及元认知数据选择，它使用类似的自我判断来识别高价值训练样本，优于朴素的主动学习。我们将这些创新应用于忠实校准（FC）问题，该任务本身本质上是元认知的：目标是使表达的不确定性与内在不确定性对齐，即使对于前沿LLM也很困难。我们采用两阶段解耦方法，首先使用这些方法校准模型自我报告置信度分数的忠实性，然后通过定向输出编辑映射到自然、上下文可适应的语言不确定性。大量实验表明，RLMF在保持准确性的同时，在多种任务上实现了可泛化的、最先进的FC。此外，RLMF比标准RL提升高达63%，同时增强了模型评估和表达自身能力极限的能力。这使得RLMF成为增强LLM元认知以提升能力和对齐的有前景范式，并表明元认知表现可作为有效的RL信号，克服先前内在反馈方法的局限性。
英文摘要：
Metacognition is a critical component of intelligence that describes the ability to monitor and regulate one's own cognitive processes. Yet LLMs exhibit systemic deficiencies in key metacognitive faculties: they hallucinate with high confidence, fail to recognize knowledge boundaries, and misrepresent their internal uncertainty--undermining trustworthiness and reliability. Since monitoring task performance and adapting behavior accordingly are central to metacognition, we posit that models capable of accurately judging their own performance are better positioned to improve it. We operationalize this idea via two novel mechanisms: reinforcement learning with metacognitive feedback (RLMF), a paradigm to refine completion rankings during preference optimization based on the quality of a model's self-judgments of performance, and metacognitive data selection, which uses similar self-judgments to identify high-value training examples, outperforming naive active learning. We apply these innovations to the problem of faithful calibration (FC), a task that is itself fundamentally metacognitive: the goal is to align expressed with intrinsic uncertainty, difficult even for frontier LLMs. We adopt a two-stage, decoupled approach, first using these methods to calibrate the faithfulness of models' self-reported confidence scores, then mapping to natural, context-adaptable linguistic uncertainty via targeted output editing. Extensive experiments show RLMF achieves generalizable, state-of-the-art FC on diverse tasks while preserving accuracy. Further, RLMF surpasses standard RL by up to 63% while enhancing models' ability to assess and express their own capability limits. This positions RLMF as a promising paradigm to enhance LLM metacognition toward improved abilities and alignment, and suggests metacognitive performance as an effective RL signal to overcome limits of prior intrinsic feedback methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
