HERO: 基于环境观察的后见增强反思的智能体自蒸馏
HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation
AI总结：
提出HERO框架，利用环境观察作为局部对齐反馈进行自蒸馏，解决多轮设置中特权反馈与当前决策上下文不对齐导致的性能下降问题，在TauBench和WebShop上提升任务成功率并减少冗余轮次。
摘要：
强化学习通常通过轨迹的终端结果来提升多轮智能体能力，这使得难以确定每个中间轮的信用分配。最近的在线自蒸馏方法通过自教师将特权反馈转化为密集的令牌级监督，提供了一种有前景的替代方案。我们的研究动机是观察到当朴素地将此范式扩展到多轮设置时出现意外的性能下降，我们将其归因于特权反馈（如成功轨迹或终端结果）与学生当前决策上下文之间缺乏对齐。我们引入了HERO，一种后见增强的自蒸馏框架，它使用下一个环境观察作为局部对齐反馈。每次轨迹展开后，HERO反思完成的交互，将每个观察转化为紧凑的轮级诊断，捕获关于原始动作的可操作反馈，如其必要性、有效性或失败原因。在TauBench和WebShop上，HERO比仅环境反馈的自蒸馏和GRPO提高了任务成功率并减少了不必要的轮次。在训练轮次预算有限（成功轨迹稀少且GRPO提供弱奖励对比信号）的情况下，它尤其有效。
英文摘要：
Reinforcement learning typically improves multi-turn agent capabilities through the terminal outcome of the trajectories, which makes it difficult to determine credit assignments for each intermediate turns. Recent on-policy self-distillation methods offer a promising alternative by converting privileged feedback into dense token-level supervision through a self-teacher. Our study is motivated by the unexpected performance degradation observed when naively extending this paradigm to multi-turn settings, which we attribute to a lack of alignment between privileged feedback, such as successful trajectories or terminal outcomes, and the student's current decision context. We introduce HERO, a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback. After each rollout, HERO reflects on the completed interaction to convert each observation into a compact turn-level diagnosis, that captures actionable feedback about the original action such as its necessity, validity or failure cause. On TauBench and WebShop, HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO. It is especially effective under limited training turn budgets, where successful rollouts are rare and GRPO provides weak reward-contrast signals.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
