基于在线演示的机器人策略迁移：一种主动强化学习方法
Robot Policy Transfer with Online Demonstrations: An Active Reinforcement Learning Approach
AI总结：
本文提出主动LfD算法Policy Transfer with Online Demonstrations，在有限演示预算下优化在线专家演示查询的时机与内容，显著提升机器人跨环境、任务和实体的策略迁移性能。
摘要：
迁移学习（TL）是一种强大的工具，使机器人能够在不同环境、任务或实体之间迁移已学习的策略。为进一步促进这一过程，已有研究尝试将其与演示学习（LfD）相结合，以实现更灵活、更高效的策略迁移。然而，这些方法几乎完全局限于在策略迁移开始前收集的离线演示，可能受到LfD所带来的协变量偏移这一固有问题的影响，并损害策略迁移的性能。与此同时，从零开始学习设置中的大量研究表明，在线演示能够有效缓解协变量偏移，并以更高的样本效率带来更好的策略性能。本工作结合这些洞见，将在线演示引入策略迁移设置。我们提出了Policy Transfer with Online Demonstrations，这是一种用于策略迁移的主动LfD算法，能够在有限的演示预算下，优化对在线回合制专家演示进行查询的时机和内容。我们在八个机器人场景中评估了该方法，涉及跨不同环境特征、任务目标和机器人实体的策略迁移，目标是将已训练策略从源任务迁移到相关但不同的目标任务。结果表明，与两种采用离线演示的经典LfD方法和一种采用在线演示的主动LfD方法相比，我们的方法在平均成功率和样本效率方面均显著优于所有基线。此外，我们还在真实环境中的三个迁移场景下对迁移后策略进行了初步的仿真到真实测试，证明了该策略在真实机械臂上的有效性。
英文摘要：
Transfer Learning (TL) is a powerful tool that enables robots to transfer learned policies across different environments, tasks, or embodiments. To further facilitate this process, efforts have been made to combine it with Learning from Demonstrations (LfD) for more flexible and efficient policy transfer. However, these approaches are almost exclusively limited to offline demonstrations collected before policy transfer starts, which may suffer from the intrinsic issue of covariance shift brought by LfD and harm the performance of policy transfer. Meanwhile, extensive work in the learning-from-scratch setting has shown that online demonstrations can effectively alleviate covariance shift and lead to better policy performance with improved sample efficiency. This work combines these insights to introduce online demonstrations into a policy transfer setting. We present Policy Transfer with Online Demonstrations, an active LfD algorithm for policy transfer that can optimize the timing and content of queries for online episodic expert demonstrations under a limited demonstration budget. We evaluate our method in eight robotic scenarios, involving policy transfer across diverse environment characteristics, task objectives, and robotic embodiments, with the aim to transfer a trained policy from a source task to a related but different target task. The results show that our method significantly outperforms all baselines in terms of average success rate and sample efficiency, compared to two canonical LfD methods with offline demonstrations and one active LfD method with online demonstrations. Additionally, we conduct preliminary sim-to-real tests of the transferred policy on three transfer scenarios in the real-world environment, demonstrating the policy effectiveness on a real robot manipulator.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
