利用外在行为好奇心实现策略行为多样化
Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity
AI总结：
针对模仿学习局限于单一策略的问题，提出质量多样性逆强化学习框架QD-IRL，并引入外在行为好奇心EBC机制，通过外部评论家提供的好奇心奖励引导智能体探索多样化行为，显著提升了多种机器人移动任务的性能。
摘要：
模仿学习（IL）在多种应用（如机器人移动）中展现出潜力，但通常局限于学习单一专家策略，这限制了在不可预测的现实场景中的行为多样性和鲁棒性。为解决此问题，我们提出了质量多样性逆强化学习（QD-IRL），这是一个将质量多样性优化与IL方法相结合的新颖框架，使智能体能够从有限的演示中学习多样化行为。本工作引入了外在行为好奇心（EBC），它允许智能体根据行为相对于大型行为档案的新颖性，从外部评论家处获得额外的好奇心奖励。为验证EBC在探索多样化移动行为方面的有效性，我们在多个机器人移动任务上评估了该方法。在所有包含的环境中，EBC使基于GAIL、VAIL和DiffAIL的QD-IRL实例性能分别提升了高达185%、42%和150%，在Humanoid中甚至超越了专家性能20%。此外，我们证明EBC适用于基于梯度树的质量多样性强化学习（QD-RL）算法，它显著提升了性能，并为学习行为多样化策略提供了通用技术。本工作的源代码已在 https://github.com/vanzll/EBC 提供。
英文摘要：
Imitation learning (IL) has shown promise in various applications (e.g. robot locomotion) but is often limited to learning a single expert policy, constraining behavior diversity and robustness in unpredictable real-world scenarios. To address this, we introduce Quality Diversity Inverse Reinforcement Learning (QD-IRL), a novel framework that integrates quality-diversity optimization with IRL methods, enabling agents to learn diverse behaviors from limited demonstrations. This work introduces Extrinsic Behavioral Curiosity (EBC), which allows agents to receive additional curiosity rewards from an external critic based on how novel the behaviors are with respect to a large behavioral archive. To validate the effectiveness of EBC in exploring diverse locomotion behaviors, we evaluate our method on multiple robot locomotion tasks. EBC improves the performance of QD-IRL instances with GAIL, VAIL, and DiffAIL across all included environments by up to 185%, 42%, and 150%, even surpassing expert performance by 20% in Humanoid. Furthermore, we demonstrate that EBC is applicable to Gradient-Arborescence-based Quality Diversity Reinforcement Learning (QD-RL) algorithms, where it substantially improves performance and provides a generic technique for learning behavioral-diverse policies. The source code of this work is provided at https://github.com/vanzll/EBC.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
