Equivariant neural networks enforce symmetry within the structure of their convolutional layers, resulting in a substantial improvement in sample efficiency when learning an equivariant or invariant function. Such models are applicable to robotic manipulation learning which can often be formulated as a rotationally symmetric problem. This paper studies equivariant model architectures in the context of $Q$-learning and actor-critic reinforcement learning. We identify equivariant and invariant characteristics of the optimal $Q$-function and the optimal policy and propose equivariant DQN and SAC algorithms that leverage this structure. We present experiments that demonstrate that our equivariant versions of DQN and SAC can be significantly more sample efficient than competing algorithms on an important class of robotic manipulation problems.
Khoury College of Computer Sciences, Northeastern University(东北大学卡尔霍里计算机科学学院);Department of Electrical and Computer Engineering, Northeastern University(东北大学电气与计算机工程系)
Partially Equivariant Reinforcement Learning in Symmetry-Breaking Environments
School of Mechanical Engineering, Yonsei University, Seoul, South Korea(延世大学机械工程学院,首尔,韩国);Department of Artificial Intelligence, Yonsei University, Seoul, South Korea(延世大学人工智能系,首尔,韩国);Department of Mechanical Engineering, University of California, Berkeley, United States(加州大学伯克利分校机械工程系,美国)