arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.18951cs.LGcs.AIcs.RO
收藏

3D环境中的子等变图强化学习

Subequivariant Graph Reinforcement Learning in 3D Environments

Tsinghua University(清华大学);THU-Bosch JCML Center(清华-博世机器学习联合中心);Renmin University of China(中国人民大学)

Runfa Chen, Jiaqi Han, Fuchun Sun, Wenbing Huang

AI总结:

本文提出3D-SGRL框架,通过引入3D形态无关RL基准和子等变Transformer注入几何对称性,使策略在任意方向泛化,并在单任务、多任务及零样本场景中显著优于现有方法。

摘要:

学习一个共享策略来引导不同智能体的运动是强化学习(RL)中的核心兴趣之一,这引出了对形态无关RL的研究。然而,现有基准在起点和目标点的选择上具有高度限制性,将智能体的运动约束在2D空间内。在这项工作中,我们提出了一种新颖的形态无关RL设置,称为3D环境中的子等变图RL(3D-SGRL)。具体来说,我们首先在3D空间中引入一组新的更实用但更具挑战性的基准,允许智能体拥有完整的自由度,从任意配置出发沿任意方向探索。此外,为了在扩大的状态-动作空间上优化策略,我们提出将几何对称性,即子等变性,注入到策略和Q函数的建模中,使得策略能够泛化到所有方向,从而提高探索效率。这一目标通过一种新颖的子等变Transformer(SET)实现,该Transformer允许表达性的消息交换。最后,我们在所提出的基准上评估了所提出的方法,我们的方法在单任务、多任务和零样本泛化场景中一致且显著地优于现有方法。我们还进行了大量消融实验以验证我们的设计。代码和视频可在我们的项目页面获取:https://alpc91.github.io/SGRL/。

英文摘要:

Learning a shared policy that guides the locomotion of different agents is of core interest in Reinforcement Learning (RL), which leads to the study of morphology-agnostic RL. However, existing benchmarks are highly restrictive in the choice of starting point and target point, constraining the movement of the agents within 2D space. In this work, we propose a novel setup for morphology-agnostic RL, dubbed Subequivariant Graph RL in 3D environments (3D-SGRL). Specifically, we first introduce a new set of more practical yet challenging benchmarks in 3D space that allows the agent to have full Degree-of-Freedoms to explore in arbitrary directions starting from arbitrary configurations. Moreover, to optimize the policy over the enlarged state-action space, we propose to inject geometric symmetry, i.e., subequivariance, into the modeling of the policy and Q-function such that the policy can generalize to all directions, improving exploration efficiency. This goal is achieved by a novel SubEquivariant Transformer (SET) that permits expressive message exchange. Finally, we evaluate the proposed method on the proposed benchmarks, where our method consistently and significantly outperforms existing approaches on single-task, multi-task, and zero-shot generalization scenarios. Extensive ablations are also conducted to verify our design. Code and videos are available on our project page: https://alpc91.github.io/SGRL/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑