Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?
National University of Defense Technology(国防科技大学);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);Hebei University of Technology(河北工业大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区)
Yang Dai, Oubo Ma, Longfei Zhang, Xingxing Liang, Shengchao Hu, Mengzhu Wang, Shouling Ji, Jincai Huang, Li Shen
Transformer-based trajectory optimization methods have demonstrated exceptional performance in offline Reinforcement Learning (offline RL). Yet, it poses challenges due to substantial parameter size and limited scalability, which is particularly critical in sequential decision-making scenarios where resources are constrained such as in robots and drones with limited computational power. Mamba, a promising new linear-time sequence model, offers performance on par with transformers while delivering substantially fewer parameters on long sequences. As it remains unclear whether Mamba is compatible with trajectory optimization, this work aims to conduct comprehensive experiments to explore the potential of Decision Mamba (dubbed DeMa) in offline RL from the aspect of data structures and essential components with the following insights: (1) Long sequences impose a significant computational burden without contributing to performance improvements since DeMa's focus on sequences diminishes approximately exponentially. Consequently, we introduce a Transformer-like DeMa as opposed to an RNN-like DeMa. (2) For the components of DeMa, we identify the hidden attention mechanism as a critical factor in its success, which can also work well with other residual structures and does not require position embedding. Extensive evaluations demonstrate that our specially designed DeMa is compatible with trajectory optimization and surpasses previous methods, outperforming Decision Transformer (DT) with higher performance while using 30\% fewer parameters in Atari, and exceeding DT with only a quarter of the parameters in MuJoCo.
Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning
Northwestern University(西北大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Zhejiang University(浙江大学);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心)
QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);Institute of Artificial Intelligence(人工智能研究院);Robotics, Xi'an Jiaotong University(机器人技术,西安交通大学);Computer Science, University College London(计算机科学,伦敦大学学院);School of Engineering, Westlake University(工程学院,西湖大学)
Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院);School of Engineering, Great Bay University(大湾区大学工程学院)
Automatic Reward Shaping from Confounded Offline Data
Causal AI Lab, Columbia University, New York, USA(因果AI实验室,哥伦比亚大学,纽约,美国);Department of Electrical Engineering and Computer Science, Syracuse University, New York, USA(电气工程与计算机科学系, Syracuse大学,纽约,美国)
Safe Deployment of Offline Reinforcement Learning via Input Convex Action Correction
Department of Chemical Engineering, Imperial College London(帝国理工学院化学工程系);Shell Information Technology International BV(壳牌信息科技国际有限公司);Shell Global Solutions International BV(壳牌全球解决方案国际有限公司)
Off-Policy Confidence Interval Estimation with Confounded Markov Decision Process
London School of Economics and Political Science(伦敦政治经济学院);Sun Yat-sen University(中山大学);Didi Chuxing(滴滴出行);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);North Carolina State University(北卡罗来纳州立大学)
Towards Fast Safe Online Reinforcement Learning via Policy Finetuning
School of Electrical, Computer and Energy Engineering(电气、计算机与能源工程学院);Arizona State University(亚利桑那州立大学);School of Electrical Engineering and Computer Science(电气工程与计算机科学学院);Washington State University(华盛顿州立大学);Department of Computer Science(计算机科学系);University of Houston(休斯顿大学)
Quantile-Coupled Flow Matching for Distributional Reinforcement Learning
Oxford Robotics Institute, University of Oxford(牛津大学机器人研究所);Bristol Robotics Laboratory, University of the West of England(布里斯托尔机器人实验室,西英格兰大学);Dyson Institute of Engineering & Technology(戴森工程与技术研究院)