端到端强化学习提升灵巧抓取策略
End-to-end RL Improves Dexterous Grasping Policies
AI总结：
本文针对灵巧抓取中视觉端到端强化学习批大小受限的问题，提出将仿真器与PPO训练拆分到不同GPU的方法，使环境数量翻倍，并通过深度策略蒸馏到双目RGB网络，在仿真和真实世界中均超越此前最先进的视觉抓取结果。
摘要：
本文探索了在机械臂+灵巧手系统中扩展基于图像的端到端学习以完成灵巧抓取的技术。与基于状态的强化学习（RL）相比，基于视觉的强化学习在内存效率上要低得多，导致批大小相对较小，这对PPO等算法并不友好。尽管如此，端到端RL仍然是一种有吸引力的方法，因为与更常用的将基于状态的策略蒸馏到视觉网络中的技术不同，端到端RL可以催生主动视觉等涌现行为。我们发现训练这些策略的一个关键瓶颈在于，大多数现有仿真器使用传统数据并行技术扩展到多个GPU的方式。我们提出了一种新方法，将仿真器与RL（包括训练和经验缓冲区）拆分到不同的GPU上。在一个配备四块GPU的节点上，我们让仿真器运行在其中三块上，PPO运行在第四块上。我们证明，在GPU数量相同的情况下，与之前标准数据并行的基线相比，我们能够将现有环境数量翻倍。这使我们能够端到端地训练基于视觉（含深度信息）的环境，而此前这些环境在基线下表现要差得多。我们训练并将基于深度和基于状态的策略蒸馏到双目RGB网络中，结果表明深度蒸馏在仿真和现实中都能带来更好的结果。这一改进很可能源于状态策略与视觉策略之间的可观测性差距，而在将深度策略蒸馏到双目RGB时不存在这种差距。我们进一步表明，由拆分式仿真带来的更大批大小也提升了真实世界性能。在真实世界部署时，我们使用端到端策略超越了此前最先进的基于视觉的结果。
英文摘要：
This work explores techniques to scale up image-based end-to-end learning for dexterous grasping with an arm + hand system. Unlike state-based RL, vision-based RL is much more memory inefficient, resulting in relatively low batch sizes, which is not amenable for algorithms like PPO. Nevertheless, it is still an attractive method as unlike the more commonly used techniques which distill state-based policies into vision networks, end-to-end RL can allow for emergent active vision behaviors. We identify a key bottleneck in training these policies is the way most existing simulators scale to multiple GPUs using traditional data parallelism techniques. We propose a new method where we disaggregate the simulator and RL (both training and experience buffers) onto separate GPUs. On a node with four GPUs, we have the simulator running on three of them, and PPO running on the fourth. We are able to show that with the same number of GPUs, we can double the number of existing environments compared to the previous baseline of standard data parallelism. This allows us to train vision-based environments, end-to-end with depth, which were previously performing far worse with the baseline. We train and distill both depth and state-based policies into stereo RGB networks and show that depth distillation leads to better results, both in simulation and reality. This improvement is likely due to the observability gap between state and vision policies which does not exist when distilling depth policies into stereo RGB. We further show that the increased batch size brought about by disaggregated simulation also improves real world performance. When deploying in the real world, we improve upon the previous state-of-the-art vision-based results using our end-to-end policies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
