重新思考奖励监督：基于评分准则的自蒸馏
Rethinking Reward Supervision: Rubric-Conditioned Self-Distillation
AI总结：
提出评分准则条件自蒸馏框架，通过结构化细粒度反馈指导推理模型，在科学推理基准上平均超越GRPO 1.0分、OPSD 0.9分。
摘要：
推理语言模型的后训练通常由监督蒸馏和基于可验证奖励的强化学习驱动。蒸馏通常依赖于思维链注释，这些注释获取成本高昂，且可能本身带有噪声、不完整或部分错误；即使最终答案正确，不完美的推理过程也会干扰学习。另一方面，基于验证奖励的强化学习通常将评估反馈压缩为标量信号，掩盖了响应中哪些方面需要改进。我们提出\textbf{评分准则条件自蒸馏}框架，该框架将评分准则作为结构化、细粒度的反馈用于策略内自蒸馏。我们的方法使教师模型以准则级评分准则为条件，并利用它在学生自身采样的轨迹上提供令牌级指导。这种设计避免了将单一参考推理过程作为唯一的监督目标。相反，评分准则指定了一个强响应应满足的条件，从而在推理过程中实现比标量奖励优化更细粒度的信用分配。我们通过一个两阶段流程实例化该框架：首先学习生成任务特定的评分准则，然后训练一个评分准则引导的推理器。我们在多样化的科学推理基准上进行评估，结果表明，评分准则条件自蒸馏有效地将准则级标准转化为推理过程中的令牌级指导，平均超过GRPO 1.0分、OPSD 0.9分。
英文摘要：
Post-training of reasoning language models is commonly driven by supervised distillation and reinforcement learning with verifiable rewards. Distillation often relies on chain-of-thought annotations that are expensive to obtain and may themselves be noisy, incomplete, or partially incorrect; even when the final solution is correct, an imperfect rationale can interfere with learning. Reinforcement learning with verified rewards, on the other hand, typically compresses evaluative feedback into a scalar signal, obscuring which aspects of a response should be improved. We propose \textbf{Rubric-Conditioned Self-Distillation}, a framework that incorporates rubrics as structured, fine-grained feedback for on-policy self-distillation. Our method conditions the teacher model on criterion-level rubrics and uses it to provide token-level guidance on the student's own sampled trajectories. This design avoids treating a single reference rationale as the sole supervision target. Instead, rubrics specify what a strong response should satisfy, enabling more fine-grained credit assignment over the reasoning process than scalar reward optimization. We instantiate this framework with a two-stage pipeline that first learns to generate task-specific rubrics and then trains a rubric-guided reasoner. We evaluate on a diverse suite of science reasoning benchmarks and results show that rubric-conditioned self-distillation effectively converts rubric-level criteria into token-level guidance over the reasoning process, surpassing GRPO by 1.0 points and OPSD by 0.9 points on average.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
