arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2410.09536cs.LGcs.RO
收藏

TOP-ERL:基于 Transformer 的离策略情景强化学习

TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learning

Karlsruhe Institute of Technology(卡尔斯鲁厄理工学院)

Ge Li, Dong Tian, Hongyi Zhou, Xinkai Jiang, Rudolf Lioutikov, Gerhard Neumann

AI总结:

针对 ERL 难以评估完整动作序列价值、只能在策略训练的问题,提出 TOP-ERL,通过轨迹分段、Transformer critic 与 n-step 回报实现稳定离策略更新,并在机器人学习任务中显著超越现有 RL 方法。

摘要:

本工作提出了基于 Transformer 的离策略情景强化学习(Transformer-based Off-Policy Episodic Reinforcement Learning,TOP-ERL),这是一种能够在 ERL 框架中实现离策略更新的新型算法。在 ERL 中,策略预测跨多个时间步的完整动作轨迹,而不是在每个时间步预测单个动作。这些轨迹通常由运动基元(Movement Primitives,MP)等轨迹生成器参数化,从而能够在长时域上进行平滑、高效的探索,同时捕捉高层时间相关性。然而,由于难以评估完整动作序列的状态-动作价值,ERL 方法往往受限于在策略框架,限制了样本效率,并阻碍了更高效离策略架构的使用。TOP-ERL 通过切分长动作序列,并利用基于 Transformer 的 critic 架构结合 n-step 回报估计来估计每个片段的状态-动作价值,从而解决了这一缺陷。这些贡献带来了高效且稳定的训练,并反映在复杂机器人学习环境中开展的实验结果上。TOP-ERL 显著优于最先进的 RL 方法。详尽的消融研究还展示了关键设计选择对模型性能的影响。

英文摘要:

This work introduces Transformer-based Off-Policy Episodic Reinforcement Learning (TOP-ERL), a novel algorithm that enables off-policy updates in the ERL framework. In ERL, policies predict entire action trajectories over multiple time steps instead of single actions at every time step. These trajectories are typically parameterized by trajectory generators such as Movement Primitives (MP), allowing for smooth and efficient exploration over long horizons while capturing high-level temporal correlations. However, ERL methods are often constrained to on-policy frameworks due to the difficulty of evaluating state-action values for entire action sequences, limiting their sample efficiency and preventing the use of more efficient off-policy architectures. TOP-ERL addresses this shortcoming by segmenting long action sequences and estimating the state-action values for each segment using a transformer-based critic architecture alongside an n-step return estimation. These contributions result in efficient and stable training that is reflected in the empirical results conducted on sophisticated robot learning environments. TOP-ERL significantly outperforms state-of-the-art RL methods. Thorough ablation studies additionally show the impact of key design choices on the model performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑