元认知作为奖励：通过知识和调节信号强化LLM推理
Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals
AI总结：
提出元认知奖励（MaR）框架，利用元认知知识和调节信号作为过程奖励，提升LLM推理质量，在22个基准上平均提升7.7%。
摘要：
最近的强化学习方法显著提高了LLM的推理能力。现有的奖励设计主要遵循两种范式：(1) 基于可验证奖励的强化学习（RLVR）从可执行检查或真实答案中获取结果信号，但对中间推理行为的指导有限。(2) 基于评分标准的奖励（RaR）通过使用自然语言评分标准来评估推理质量和任务合规性，超越了最终答案检查，但通常需要实例特定的评分标准和大量设计工作。为解决这些问题，我们引入了元认知奖励（MaR），一种受元认知启发的RL框架，通过两个通用过程维度指导LLM推理：i) 元认知知识，无需手工制作的实例特定评分标准即可识别任务相关信息；ii) 元认知调节，规划和调整推理过程，以提供超越最终答案结果的奖励指导。MaR将模型轨迹分解为显式的元认知组件，并通过任务知识覆盖度、调节保真度和最终答案正确性的轨迹级奖励进行优化。通过这种方式，MaR将奖励反馈扩展到推理轨迹，同时将奖励信号锚定在通用的元认知维度上。在22个基准上的实验表明，MaR持续提升模型性能，相比基础模型最高提升7.7%，相比原始DAPO最高提升11.0%。值得注意的是，Qwen3.5-9B + MaR缩小了与前沿模型的差距，在整体平均上超越GPT-OSS-120B，并在多个单独基准上超越更强模型。过程级分析进一步显示推理过程质量显著提升。MaR还能泛化到域外数据集，MaR训练的模型在平均性能上优于对应的基础模型。
英文摘要：
Recent RL methods have substantially improved the reasoning abilities of LLMs. Existing reward designs mainly follow two paradigms: (1) Reinforcement learning with verifiable rewards (RLVR) derives outcome signals from executable checks or ground-truth answers, but provides limited guidance for intermediate reasoning behaviors. (2) Rubrics-as-reward (RaR) goes beyond final-answer checking by using natural-language rubrics to assess reasoning quality and task compliance, but often requires instance-specific rubrics and substantial design effort. To address these issues, we introduce Metacognition-as-Reward (MaR), a metacognition-inspired RL framework that guides LLM reasoning through two general process dimensions: i) metacognitive knowledge, which identifies task-relevant information without hand-crafted instance-specific rubrics, and ii) metacognitive regulation, which plans and adjusts the reasoning process to provide reward guidance beyond final-answer outcomes. MaR scaffolds model rollouts into explicit metacognitive components and optimizes them with a trajectory-level reward over task knowledge coverage, regulation fidelity, and final-answer correctness. In this way, MaR extends reward feedback to reasoning trajectories while grounding the reward signals in general metacognitive dimensions. Experiments on 22 benchmarks show that MaR consistently improves model performance, achieving up to a 7.7% gain over the base model and up to an 11.0% gain over vanilla DAPO. Notably, Qwen3.5-9B + MaR narrows the gap to frontier models, surpassing GPT-OSS-120B on overall average and outperforming stronger models on several individual benchmarks. Process-level analysis further shows substantial improvements in reasoning process quality. MaR also generalizes to out-of-domain datasets, where MaR-trained models improve over their corresponding base models on average.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
