自蒸馏代理强化学习
Self-Distilled Agentic Reinforcement Learning
AI总结：
本文提出SDAR方法，通过将OPSD作为辅助目标并保持RL为主优化，解决多轮代理中的稳定性问题，提升性能。
摘要：
强化学习（RL）已逐渐成为训练后LLM代理的核心范式，但其轨迹级奖励信号仅能为长周期交互提供粗略监督。在策略上自我蒸馏（OPSD）通过引入一个增强的特权上下文的教师分支，提供密集的token级指导。然而，将OPSD转移到多轮代理时存在问题：复合多轮不稳定性会破坏监督，而基于技能的特权指导可能需要对负向教师拒绝进行不对称处理，这可能源于技能检索或利用的不完美。我们提出SDAR（自我蒸馏代理强化学习），将OPSD作为门控辅助目标，同时保持RL作为主要优化核心。SDAR将分离的token级信号映射到sigmoid门，强化对教师认可的正向缺口token的蒸馏，并温和地减弱负向教师拒绝。在ALFWorld、WebShop和Search-QA上，SDAR在Qwen2.5和Qwen3家族中显著优于GRPO（在ALFWorld上提升9.4%，在Search-QA上提升7.0%，在WebShop-Acc上提升10.2%），避免了朴素GRPO+OPSD的不稳定性，并在不同模型规模上一致优于混合RL-OPSD基线。
英文摘要：
Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
