基于GPU加速仿真的跨机器人任务种群强化学习基准测试
Benchmarking Population-Based Reinforcement Learning across Robotic Tasks with GPU-Accelerated Simulation
AI总结：
针对深度强化学习超参数敏感、训练成本高的问题，本研究结合PBRL与GPU加速仿真，在四类机器人任务上验证其性能优于主流基线算法，首次实现PBRL智能体的仿真到现实部署。
摘要：
近年来，深度强化学习（RL）在解决复杂连续控制任务上展现出有效性，但这需要海量训练经验作为代价，学习效率与策略性能对超参数选择的敏感性进一步加剧了该问题，往往需要开展大量耗时的反复试验。本研究采用种群强化学习（PBRL）方法与GPU加速物理仿真器，通过并行训练多个策略提升RL的探索能力。研究将PBRL框架与PPO、SAC、DDPG三种当前最优RL算法开展基准对比，该框架可根据学习智能体的性能动态调整超参数。实验在Isaac Gym的四个高难度任务——Anymal Terrain、Shadow Hand、Humanoid、Franka Nut Pick上开展，分析了种群规模与超参数变异机制的影响。结果显示，在累计奖励指标上，PBRL智能体的性能优于非进化基线智能体。此外，训练后的智能体最终被部署到真实世界执行Franka Nut Pick任务，据我们所知，这是首次将PBRL智能体部署到真实硬件的 sim-to-real（仿真到现实）尝试。所学策略的代码与视频可在项目网站（https://sites.google.com/view/pbrl）获取。
英文摘要：
In recent years, deep reinforcement learning (RL) has shown its effectiveness in solving complex continuous control tasks. However, this comes at the cost of an enormous amount of experience required for training, exacerbated by the sensitivity of learning efficiency and the policy performance to hyperparameter selection, which often requires numerous trials of time-consuming experiments. This work leverages a Population-Based Reinforcement Learning (PBRL) approach and a GPU-accelerated physics simulator to enhance the exploration capabilities of RL by concurrently training multiple policies in parallel. The PBRL framework is benchmarked against three state-of-the-art RL algorithms -- PPO, SAC, and DDPG -- dynamically adjusting hyperparameters based on the performance of learning agents. The experiments are performed on four challenging tasks in Isaac Gym -- Anymal Terrain, Shadow Hand, Humanoid, Franka Nut Pick -- by analyzing the effect of population size and mutation mechanisms for hyperparameters. The results show that PBRL agents achieve superior performance, in terms of cumulative reward, compared to non-evolutionary baseline agents. Moreover, the trained agents are finally deployed in the real world for a Franka Nut Pick task. To our knowledge, this is the first sim-to-real attempt for deploying PBRL agents on real hardware. Code and videos of the learned policies are available on our project website (https://sites.google.com/view/pbrl).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
