优先轨迹回放：一种面向数据驱动强化学习的回放记忆机制
Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning
AI总结：
针对离线强化学习中状态转移采样技术效果不稳定的问题，提出轨迹回放与优先轨迹回放机制，通过轨迹层面采样及优先级排序提升学习效率，在D4RL上验证了其对现有离线RL算法的性能提升作用。
摘要：
近年来，数据驱动强化学习（RL，也称为离线强化学习）受到了广泛关注。然而，尽管数据采样技术具备提升在线强化学习性能的潜力，其在离线强化学习中的作用却一直被忽视。近期研究表明，在离线强化学习中直接对状态转移应用采样技术并不能持续提升性能。因此，本研究提出了一种记忆机制——（优先）轨迹回放（Trajectory Replay / Prioritized Trajectory Replay，简称TR/PTR），该方法将采样视角拓展至轨迹层面，以从有限数据中提取更全面的信息。TR通过轨迹反向采样优化后续状态信息的利用，从而提升学习效率。在TR的基础上，我们构建了加权 critic 目标以避免在离线训练中采样未见过的动作，还提出了优先轨迹回放（PTR），该方法可通过多种轨迹优先级指标进行优先级排序，实现更高效的轨迹采样。我们在D4RL数据集上验证了将TR和PTR与现有离线强化学习算法集成的优势。综上，本研究强调了基于轨迹的数据采样技术在提升离线强化学习算法效率与性能方面的重要意义。
英文摘要：
In recent years, data-driven reinforcement learning (RL), also known as offline RL, have gained significant attention. However, the role of data sampling techniques in offline RL has been overlooked despite its potential to enhance online RL performance. Recent research suggests applying sampling techniques directly to state-transitions does not consistently improve performance in offline RL. Therefore, in this study, we propose a memory technique, (Prioritized) Trajectory Replay (TR/PTR), which extends the sampling perspective to trajectories for more comprehensive information extraction from limited data. TR enhances learning efficiency by backward sampling of trajectories that optimizes the use of subsequent state information. Building on TR, we build the weighted critic target to avoid sampling unseen actions in offline training, and Prioritized Trajectory Replay (PTR) that enables more efficient trajectory sampling, prioritized by various trajectory priority metrics. We demonstrate the benefits of integrating TR and PTR with existing offline RL algorithms on D4RL. In summary, our research emphasizes the significance of trajectory-based data sampling techniques in enhancing the efficiency and performance of offline RL algorithms.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
