arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2405.12094cs.LG
收藏

Mamba 与离线强化学习中的轨迹优化兼容吗?

Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?

National University of Defense Technology(国防科技大学);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);Hebei University of Technology(河北工业大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区)

Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, Li Shen

AI总结:

本文研究 Mamba 能否用于离线 RL 轨迹优化,提出类似 Transformer 的 DeMa 并验证隐藏注意力的关键作用,以更少参数在 Atari 和 MuJoCo 上超过 DT。

摘要:

基于 Transformer 的轨迹优化方法已在离线强化学习(offline RL)中展现出卓越性能。然而,由于参数量庞大且可扩展性有限,它带来了挑战;这在资源受限的序列决策场景中尤为关键,例如计算能力有限的机器人和无人机。Mamba 是一种颇具前景的新型线性时间序列模型,在长序列上能够提供与 Transformer 相当的性能,同时参数显著更少。由于 Mamba 是否与轨迹优化兼容仍不明确,本工作旨在开展全面实验,从数据结构和关键组件的角度探索 Decision Mamba(称为 DeMa)在离线 RL 中的潜力,并得到以下洞见:(1)长序列会造成显著的计算负担,却不会带来性能提升,因为 DeMa 对序列的关注大约呈指数级衰减。因此,我们提出一种类似 Transformer 的 DeMa,而不是类似 RNN 的 DeMa。(2)对于 DeMa 的组件,我们发现隐藏注意力机制是其成功的关键因素;该机制也能与其他残差结构良好配合,并且不需要位置嵌入。大量评估表明,我们专门设计的 DeMa 与轨迹优化兼容,并超越了以往方法:在 Atari 中以减少 30% 参数实现高于 Decision Transformer(DT)的性能,在 MuJoCo 中仅用四分之一参数便超过 DT。

英文摘要:

Transformer-based trajectory optimization methods have demonstrated exceptional performance in offline Reinforcement Learning (offline RL). Yet, it poses challenges due to substantial parameter size and limited scalability, which is particularly critical in sequential decision-making scenarios where resources are constrained such as in robots and drones with limited computational power. Mamba, a promising new linear-time sequence model, offers performance on par with transformers while delivering substantially fewer parameters on long sequences. As it remains unclear whether Mamba is compatible with trajectory optimization, this work aims to conduct comprehensive experiments to explore the potential of Decision Mamba (dubbed DeMa) in offline RL from the aspect of data structures and essential components with the following insights: (1) Long sequences impose a significant computational burden without contributing to performance improvements since DeMa's focus on sequences diminishes approximately exponentially. Consequently, we introduce a Transformer-like DeMa as opposed to an RNN-like DeMa. (2) For the components of DeMa, we identify the hidden attention mechanism as a critical factor in its success, which can also work well with other residual structures and does not require position embedding. Extensive evaluations demonstrate that our specially designed DeMa is compatible with trajectory optimization and surpasses previous methods, outperforming Decision Transformer (DT) with higher performance while using 30\% fewer parameters in Atari, and exceeding DT with only a quarter of the parameters in MuJoCo.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑