动态双粒度技能库用于智能体强化学习
Dynamic Dual-Granularity Skill Bank for Agentic RL
AI总结：
提出D2Skill，一种动态双粒度技能库，通过任务技能和步骤技能分别提供高层指导和细粒度决策支持，利用配对基线回放和技能注入回放的性能差距更新技能和优化策略，在ALFWorld等任务上显著提升性能。
摘要：
智能体强化学习可以从可重复使用的经验中显著受益，但现有的基于技能的方法主要提取轨迹级指导，并且通常缺乏维护不断演化的技能记忆的原则性机制。我们提出D2Skill，一种用于智能体强化学习的动态双粒度技能库，将可重复使用的经验组织成任务技能（用于高层指导）和步骤技能（用于细粒度决策支持和错误纠正）。D2Skill通过在同一策略下进行配对的基线回放和技能注入回放，利用它们的性能差距推导出事后效用信号，用于技能更新和策略优化。技能库完全由训练时的经验构建，通过反思不断扩展，并通过效用感知的检索和修剪进行维护。在ALFWorld、WebShop和Search-Augmented QA任务上的实验表明，D2Skill在不同规模的模型上显著优于无技能的基线。进一步的消融和分析表明，双粒度技能建模和动态技能维护对这些增益至关重要，而学习到的技能表现出更高的效用，能够跨评估设置迁移，并且仅带来适度的训练开销。
英文摘要：
Agentic RL can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld, WebShop, and Search-Augmented QA tasks show that D2Skill substantially improves performance over skill-free baselines across models of different scales. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
