正确性不足：通过执行器导向的奖励训练推理计划器
Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards
AI总结：
本文提出TraceLift框架，通过执行器导向的奖励提升推理质量，利用rubric-based Reasoning Reward Model评估推理轨迹的可靠性与有效性。
摘要：
可验证奖励的强化学习已成为提升大语言模型显式推理的常见方法，但仅凭最终答案正确性无法揭示推理轨迹的忠实性、可靠性或对消费模型的效用。为此，我们提出TraceLift，将推理视为可消费的中间产物。在计划器训练中，计划器生成标记化的推理。冻结的执行器将此推理转化为最终产物供验证器反馈，同时执行器导向的奖励塑造中间轨迹。此奖励乘以基于rubric的Reasoning Reward Model评分，乘以在相同冻结执行器上测量的提升，奖励高质量且有用的轨迹。为使推理质量直接可学习，我们引入TRACELIFT-GROUPS数据集，包含数学和代码种子问题。每个示例是同一问题组，包含高质量参考轨迹和多个可能的错误轨迹，通过局部扰动降低推理质量或解决方案支持，同时保持任务相关性。在代码和数学基准上的广泛实验表明，执行器导向的推理奖励提高了两阶段计划器-执行器系统，表明推理监督应不仅评估轨迹是否看起来好，还应评估其是否帮助消耗模型。
英文摘要：
Reinforcement learning with verifiable rewards has become a common way to improve explicit reasoning in large language models, but final-answer correctness alone does not reveal whether the reasoning trace is faithful, reliable, or useful to the model that consumes it. This outcome-only signal can reinforce traces that are right for the wrong reasons, overstate reasoning gains by rewarding shortcuts, and propagate flawed intermediate states in multi-step systems. To this end, we propose TraceLift, a planner-executor training framework that treats reasoning as a consumable intermediate artifact. During planner training, the planner emits tagged reasoning. A frozen executor turns this reasoning into the final artifact for verifier feedback, while an executor-grounded reward shapes the intermediate trace. This reward multiplies a rubric-based Reasoning Reward Model (RM) score by measured uplift on the same frozen executor, crediting traces that are both high-quality and useful. To make reasoning quality directly learnable, we introduce TRACELIFT-GROUPS, a rubric-annotated reason-only dataset built from math and code seed problems. Each example is a same-problem group containing a high-quality reference trace and multiple plausible flawed traces with localized perturbations that reduce reasoning quality or solution support while preserving task relevance. Extensive experiments on code and math benchmarks show that this executor-grounded reasoning reward improves the two-stage planner-executor system over execution-only training, suggesting that reasoning supervision should evaluate not only whether a trace looks good, but also whether it helps the model that consumes it. Our code is available at: https://github.com/MasaiahHan/TraceLift
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
