Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL
School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院);School of Engineering, Great Bay University(大湾区大学工程学院)
Qi Lv, Xiang Deng, Gongwei Chen, Michael Yu Wang, Liqiang Nie
While the conditional sequence modeling with the transformer architecture has demonstrated its effectiveness in dealing with offline reinforcement learning (RL) tasks, it is struggle to handle out-of-distribution states and actions. Existing work attempts to address this issue by data augmentation with the learned policy or adding extra constraints with the value-based RL algorithm. However, these studies still fail to overcome the following challenges: (1) insufficiently utilizing the historical temporal information among inter-steps, (2) overlooking the local intrastep relationships among return-to-gos (RTGs), states, and actions, (3) overfitting suboptimal trajectories with noisy labels. To address these challenges, we propose Decision Mamba (DM), a novel multi-grained state space model (SSM) with a self-evolving policy learning strategy. DM explicitly models the historical hidden state to extract the temporal information by using the mamba architecture. To capture the relationship among RTG-state-action triplets, a fine-grained SSM module is designed and integrated into the original coarse-grained SSM in mamba, resulting in a novel mamba architecture tailored for offline RL. Finally, to mitigate the overfitting issue on noisy trajectories, a self-evolving policy is proposed by using progressive regularization. The policy evolves by using its own past knowledge to refine the suboptimal actions, thus enhancing its robustness on noisy demonstrations. Extensive experiments on various tasks show that DM outperforms other baselines substantially.
Mamba as Decision Maker: Exploring Multi-scale Sequence Modeling in Offline Reinforcement Learning
Northwestern University(西北大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Zhejiang University(浙江大学);Beijing Innovation Center of Humanoid Robotics(北京人形机器人创新中心)
Is Mamba Compatible with Trajectory Optimization in Offline Reinforcement Learning?
National University of Defense Technology(国防科技大学);Zhejiang University(浙江大学);Shanghai Jiao Tong University(上海交通大学);Hebei University of Technology(河北工业大学);Shenzhen Campus of Sun Yat-sen University(中山大学深圳校区)
QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL
National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室);Institute of Artificial Intelligence(人工智能研究院);Robotics, Xi'an Jiaotong University(机器人技术,西安交通大学);Computer Science, University College London(计算机科学,伦敦大学学院);School of Engineering, Westlake University(工程学院,西湖大学)
Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularization
Institute for AI Industry Research (AIR), Tsinghua University(清华大学智能产业研究院(AIR));Beijing Jiaotong University(北京交通大学);UT Austin(德克萨斯大学奥斯汀分校);Tsinghua University(清华大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Latent Adversarial Regularization for Offline Preference Optimization
Department of Computer Science, Stanford University(斯坦福大学计算机科学系);Siebel School of Computing(塞贝尔计算学院);Data Science, University of Illinois at Urbana-Champaign(数据科学,伊利诺伊大学厄巴纳-香槟分校)
Offline Policy Optimization in RL with Variance Regularizaton
University of Toronto(多伦多大学);Vector Institute(向量研究所);McGill University(麦吉尔大学);Mila - Quebec AI Institute(米拉-魁北克人工智能研究所);Princeton University(普林斯顿大学);Northwestern University(西北大学);Google Research(谷歌研究院);Deepmind Montreal(深度思维蒙特利尔分所)
Offline Two-Player Zero-Sum Markov Games with KL Regularization
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);California Institute of Technology(加州理工学院);University of Virginia(弗吉尼亚大学);Purdue University(Purdue 大学);Massachusetts Institute of Technology(麻省理工学院)
Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization
JD Technology(京东科技);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Tsinghua-Berkeley Shenzhen Institute (TBSI), Tsinghua University(清华大学清华-伯克利深圳学院(TBSI));Yale University(耶鲁大学);Northwestern University(西北大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
大多数离线强化学习(RL)方法都要在改进策略以超越行为策略和约束策略以限制与行为策略的偏差之间进行权衡,因为使用分布外(OOD)动作计算$Q$值将由于分布偏移而遭受错误。最近提出的\textit{In-sample Learning}范例(即,IQL)通过只使用数据样本的分位数回归来改进策略,显示出很大的希望,因为它在不查询任何看不见的动作的价值函数的情况下学习最优策略。然而,这类方法如何处理学习价值函数的分布变化仍不清楚。在这项工作中,我们做了一个关键的发现,在样本学习范式下出现的\textit{Implicit Value Regularization}(IVR)的框架。这让我们更深入地理解了为什么样本内学习范式有效,即,它将隐式值正则化应用于策略。在IVR框架的基础上,我们进一步提出了两个实用的算法,稀疏$Q$ -学习(SQL)和指数$Q$ -学习(EQL),这两个算法采用了现有工作中使用的相同的值正则化,但在一个完整的样本内的方式。与IQL相比,我们发现我们的算法在学习价值函数时引入了稀疏性,使得它们在噪声数据体制中更加鲁棒。我们还在D4 RL基准数据集上验证了SQL和EQL的有效性,并通过将它们与小数据制度中的CQL进行比较来展示样本内学习的好处。
Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learning
School of Computer Science and Technology, Tongji University, Shanghai, China(同济大学计算机科学与技术学院,上海,中国);MOE Key Lab of Embedded System and Service Computing, Tongji University, Shanghai, China(教育部嵌入式系统与服务计算重点实验室,同济大学,上海,中国);Shanghai Innovation Institute(上海创新研究院)
Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning
National University of Defense Technology(国防科技大学);Shanghai Institute of Microsystem and Information Technology(上海微系统与信息技术研究所);Chinese Academy of Sciences(中国科学院);Delft University of Technology(代尔夫特理工大学)