快照强化学习:利用先验轨迹提升效率
Snapshot Reinforcement Learning: Leveraging Prior Trajectories for Efficiency
AI总结:
针对深度强化学习样本与计算资源需求高、现有复用方法通用性差的问题,提出仅修改环境的SnapshotRL框架及基线算法S3RL,利用教师轨迹辅助训练,在MuJoCo基准上结合多种主流算法显著提升样本效率与平均回报。
摘要:
深度强化学习(DRL)算法需要大量样本与计算资源才能实现更优性能,这限制了其实际应用,也给后续发展带来挑战。在资源有限的约束下,利用已有的计算成果(如已学习的策略、样本)来提升DRL算法的样本效率、降低计算资源消耗至关重要。现有利用已有计算成果的研究需要对现有算法和模型进行侵入式修改,且是针对特定算法专门设计的,缺乏灵活性与通用性。本文提出Snapshot Reinforcement Learning(SnapshotRL,快照强化学习)框架,该框架仅通过修改环境即可提升样本效率,无需对算法和模型做任何改动。通过允许智能体学生选择教师轨迹中的状态作为初始状态进行采样,SnapshotRL能有效利用教师轨迹辅助智能体学生训练,使其在训练早期探索更大的状态空间。我们提出了一种简单有效的SnapshotRL基线算法S3RL,该算法可与现有DRL算法良好融合。我们的实验表明,在MuJoCo基准测试上将S3RL与TD3、SAC和PPO算法结合,在无需额外样本和额外计算资源的情况下,显著提升了样本效率与平均回报。
英文摘要:
Deep reinforcement learning (DRL) algorithms require substantial samples and computational resources to achieve higher performance, which restricts their practical application and poses challenges for further development. Given the constraint of limited resources, it is essential to leverage existing computational work (e.g., learned policies, samples) to enhance sample efficiency and reduce the computational resource consumption of DRL algorithms. Previous works to leverage existing computational work require intrusive modifications to existing algorithms and models, designed specifically for specific algorithms, lacking flexibility and universality. In this paper, we present the Snapshot Reinforcement Learning (SnapshotRL) framework, which enhances sample efficiency by simply altering environments, without making any modifications to algorithms and models. By allowing student agents to choose states in teacher trajectories as the initial state to sample, SnapshotRL can effectively utilize teacher trajectories to assist student agents in training, allowing student agents to explore a larger state space at the early training phase. We propose a simple and effective SnapshotRL baseline algorithm, S3RL, which integrates well with existing DRL algorithms. Our experiments demonstrate that integrating S3RL with TD3, SAC, and PPO algorithms on the MuJoCo benchmark significantly improves sample efficiency and average return, without extra samples and additional computational resources.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。