UCOB: 通过信用感知的在策略双向自蒸馏学习利用和演化智能体技能
UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation
AI总结：
提出UCOB框架，通过比较技能条件与无技能提示的回报，实现信用感知的在策略双向自蒸馏，以有效利用和更新技能记忆，在ALFWorld、WebShop等任务上显著超越基线。
摘要：
技能记忆可以通过将过去经验作为文本指导来改进智能体强化学习，但检索到的技能并非绝对正确：它们可能在某个状态下有帮助，而在另一个状态下误导同一策略。这使得常见的特权教师假设变得脆弱，即技能条件提示可以被视为无技能提示的固定教师。我们引入了UCOB，一个通过信用感知的在策略双向自蒸馏来学习利用和演化智能体技能的框架。UCOB将技能条件提示和无技能提示视为同一模型的两个在策略上下文视图，在同一任务和锚定状态下比较它们的剩余回报，并将高回报视图作为局部教师。这种局部信用信号内化有用的技能条件行为，纠正误导性的技能使用，并指导任务/状态技能记忆更新、效用感知检索和反思自训练。在包括ALFWorld、WebShop和Search-QA的智能体任务上的实验表明，UCOB在不同模型规模上优于无技能强化学习、技能记忆基线和自蒸馏方法，在ALFWorld和WebShop上比最先进基线分别高出23.5和18.0个百分点。消融和分析进一步验证了其核心机制和效率。
英文摘要：
Skill memories can improve agentic reinforcement learning by reusing past experience as textual guidance, but retrieved skills are not oracular: they may help in one state while misleading the same policy in another. This makes the common privileged-teacher assumption fragile, namely that a skill-conditioned prompt can be treated as a fixed teacher for the no-skill prompt. We introduce UCOB, a framework for learning to utilize and evolve agentic skills via credit-aware on-policy bidirectional self-distillation. UCOB treats skill-conditioned and no-skill prompts as two on-policy context views of the same model, compares their return-to-go within the same task and anchor state, and uses the higher-return view as the local teacher. This local credit signal internalizes useful skill-conditioned behavior, corrects misleading skill usage, and guides task/state skill memory updates, utility-aware retrieval, and reflection self-training. Experiments on agentic tasks, including ALFWorld, WebShop, and Search-QA, show that UCOB outperforms skill-free RL, skill-memory baselines, and self-distillation methods across model scales, with up to 23.5 and 18.0 point gains over SOTA baselines on ALFWorld and WebShop. Ablations and analyses further validate its core mechanisms, continual adaptation across environments, and modest training overhead. Code is available at https://github.com/TU2021/UCOB.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
