反叛学生：通过自蒸馏RLVR反转教师信号进行推理探索
Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR
AI总结：
本文提出RLRT，通过反转自蒸馏信号增强正确路径上的推理，提升RLVR性能，建立信息不对称作为新设计轴。
摘要：
自蒸馏已成为训练后LLM的有效框架，其中教师通过额外信息指导学生，两者来自同一模型。虽然这种指导在学生失败时有用，但在成功轨迹中，相同机制会覆盖学生选择并抑制其推理。因此，我们提出反向阅读原始自蒸馏信号：当学生沿教师未预测的路径成功时，这些标记反映其自主推理。基于此，我们提出RLRT（RLVR与反向教师），通过GRPO增强正确轨迹上的标记。我们将其解释为RLVR中新的探索形式：非均匀多样性，而是基于学生自身成功的有价值探索。在基础、指令调优和思考调优的Qwen3检查点上，RLRT显著优于自蒸馏和基于探索的基线，确立信息不对称作为RLVR的新、原则性设计轴。
英文摘要：
Self-distillation has emerged as a powerful framework for post-training LLMs, where a teacher conditioned on extra information guides a student without it, both from the same model. While this guidance is useful when the student has failed, on successful rollouts, the same mechanism instead overwrites the student's choices and suppresses it's own reasoning. Therefore, we propose reading the original self-distillation signal in reverse: when the student succeeds along a path the teacher would not have predicted, these tokens reflect its self-driven reasoning. Building on this, we propose RLRT (RLVR with Reversed Teacher), which augments GRPO by reinforcing these tokens on correct rollouts. We interpret this as a new form of exploration in RLVR: not uniform diversity, but valuable exploration grounded in the student's own success. Across base, instruction-tuned, and thinking-tuned Qwen3 checkpoints, RLRT substantially outperforms self-distillation and exploration-based baselines, establishing information asymmetry as a new, principled design axis for RLVR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
