基于效应循环一致性的跨域策略迁移
Cross Domain Policy Transfer with Effect Cycle-Consistency
AI总结：
针对深度强化学习训练机器人策略样本效率低的问题，提出基于效应循环一致性的跨域策略迁移方法，利用非配对数据学习映射函数实现不同状态动作空间域的策略迁移，在多项任务上性能优于现有最优方法。
摘要：
由于样本效率低下，使用深度强化学习方法从零开始训练机器人策略的成本可能高得令人望而却步。为应对这一挑战，将在源域中训练的策略迁移到目标域成为一种颇具吸引力的范式。以往研究通常聚焦于状态空间和动作空间相似但其他方面存在差异的域。在本文中，我们的核心研究对象是状态空间和动作空间均不同的域，这一研究方向具有更广泛的实际意义，例如将策略从机器人A迁移到机器人B。与依赖配对数据的现有方法不同，我们提出了一种利用非配对数据学习跨域状态空间与动作空间之间映射函数的新方法。我们提出了效应循环一致性机制，通过对称优化结构对齐两个域中的转换效应，以此学习这些映射函数。一旦映射函数学习完成，我们就能将策略从源域无缝迁移到目标域。我们的方法已在3项运动任务和2项机器人操作任务上进行了测试。实验结果表明，与当前最优方法相比，我们的方法能显著降低对齐误差，并取得更优的性能。
英文摘要：
Training a robotic policy from scratch using deep reinforcement learning methods can be prohibitively expensive due to sample inefficiency. To address this challenge, transferring policies trained in the source domain to the target domain becomes an attractive paradigm. Previous research has typically focused on domains with similar state and action spaces but differing in other aspects. In this paper, our primary focus lies in domains with different state and action spaces, which has broader practical implications, i.e. transfer the policy from robot A to robot B. Unlike prior methods that rely on paired data, we propose a novel approach for learning the mapping functions between state and action spaces across domains using unpaired data. We propose effect cycle consistency, which aligns the effects of transitions across two domains through a symmetrical optimization structure for learning these mapping functions. Once the mapping functions are learned, we can seamlessly transfer the policy from the source domain to the target domain. Our approach has been tested on three locomotion tasks and two robotic manipulation tasks. The empirical results demonstrate that our method can reduce alignment errors significantly and achieve better performance compared to the state-of-the-art method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
