专家邻近度作为单次演示模仿学习的替代奖励
Expert Proximity as Surrogate Rewards for Single Demonstration Imitation Learning
AI总结：
本文针对单次演示模仿学习中奖励稀疏问题，提出基于过渡判别器的TDIL方法，通过专家状态邻近度构造密集替代奖励，在多个MuJoCo基准和机器人环境中达到专家级性能。
摘要：
本文聚焦于单次演示模仿学习（IL），这是一种适用于现实应用的方法，在这些应用中获取多个专家演示成本高昂或不可行，且真实奖励函数不可用。与具有多个演示的典型IL设置不同，单次演示IL中智能体只能访问一条专家轨迹。我们强调该设置下奖励信号稀疏的问题，并提出通过所提出的基于过渡判别器的IL（TDIL）方法来缓解这一问题。TDIL是一种逆强化学习（IRL）方法，旨在通过引入考虑环境动态的更密集替代奖励函数来解决奖励稀疏性问题。该替代奖励函数鼓励智能体向接近专家状态的状态导航。在实践中，TDIL训练一个过渡判别器来区分给定环境中的有效过渡和无效过渡，以计算替代奖励。实验表明，TDIL优于现有的IL方法，并在五个广泛采用的MuJoCo基准以及“Adroit Door”机器人环境中，在单次演示IL设置下达到了专家级性能。
英文摘要：
In this paper, we focus on single-demonstration imitation learning (IL), a practical approach for real-world applications where acquiring multiple expert demonstrations is costly or infeasible and the ground truth reward function is not available. In contrast to typical IL settings with multiple demonstrations, single-demonstration IL involves an agent having access to only one expert trajectory. We highlight the issue of sparse reward signals in this setting and propose to mitigate this issue through our proposed Transition Discriminator-based IL (TDIL) method. TDIL is an IRL method designed to address reward sparsity by introducing a denser surrogate reward function that considers environmental dynamics. This surrogate reward function encourages the agent to navigate towards states that are proximal to expert states. In practice, TDIL trains a transition discriminator to differentiate between valid and non-valid transitions in a given environment to compute the surrogate rewards. The experiments demonstrate that TDIL outperforms existing IL approaches and achieves expert-level performance in the single-demonstration IL setting across five widely adopted MuJoCo benchmarks as well as the "Adroit Door" robotic environment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
