Skill-SD：基于多轮LLM代理的技能条件自蒸馏
Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents
AI总结：
Skill-SD通过将代理自身轨迹转化为动态训练监督，结合重要加权反KL损失稳定训练，提升多轮LLM代理性能，实验显示优于标准RL和OPD方法。
摘要：
强化学习（RL）广泛用于训练多轮交互任务的LLM代理，但其样本效率受限于稀疏奖励和长 horizon。在线自蒸馏（OPSD）通过提供来自拥有真实答案的特权教师的密集标记级监督缓解了这一问题。然而，固定特权信息无法捕捉代理任务中的多样化有效策略，且简单结合OPSD与RL常导致训练崩溃。为解决这些限制，我们引入Skill-SD框架，将代理自身轨迹转化为动态训练监督。完成轨迹被总结为紧凑的自然语言技能，描述成功行为、错误和工作流程。这些技能作为动态特权信息仅条件教师，而学生始终在纯任务提示下行动，并通过蒸馏学习内化指导。为稳定训练，我们推导出重要加权反KL损失以提供梯度正确的标记级蒸馏，并动态同步教师与改进的学生。在代理基准测试中，Skill-SD显著优于标准RL基线，改进了Vanilla GRPO（在AppWorld/Sokoban上分别+14.0%/+10.9%）和Vanilla OPD（+42.1%/+40.6%）。项目页面：https://k1xe.github.io/skill-sd/
英文摘要：
Reinforcement learning (RL) has been widely used to train LLM agents for multi-turn interactive tasks, but its sample efficiency is severely limited by sparse rewards and long horizons. On-policy self-distillation (OPSD) alleviates this by providing dense token-level supervision from a privileged teacher that has access to ground-truth answers. However, such fixed privileged information cannot capture the diverse valid strategies in agent tasks, and naively combining OPSD with RL often leads to training collapse. To address these limitations, we introduce Skill-SD, a framework that turns the agent's own trajectories into dynamic training-only supervision. Completed trajectories are summarized into compact natural language skills that describe successful behaviors, mistakes, and workflows. These skills serve as dynamic privileged information conditioning only the teacher, while the student always acts under the plain task prompt and learns to internalize the guidance through distillation. To stabilize the training, we derive an importance-weighted reverse-KL loss to provide gradient-correct token-level distillation, and dynamically synchronize the teacher with the improving student. Experimental results on agentic benchmarks demonstrate that Skill-SD substantially outperforms the standard RL baseline, improving both vanilla GRPO (+14.0%/+10.9% on AppWorld/Sokoban) and vanilla OPD (+42.1%/+40.6%). Project page: https://k1xe.github.io/skill-sd/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
