arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.05427cs.LG
收藏

决策Mamba:一种具有自进化正则化的多粒度状态空间模型用于离线强化学习

Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL

School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)计算机科学与技术学院);School of Engineering, Great Bay University(大湾区大学工程学院)

Qi Lv, Xiang Deng, Gongwei Chen, Michael Yu Wang, Liqiang Nie

AI总结:

提出决策Mamba(DM),一种多粒度状态空间模型,通过Mamba架构提取时间信息、细粒度模块捕获RTG-状态-动作关系,并采用自进化正则化缓解噪声过拟合,在离线RL任务上显著超越基线。

摘要:

虽然基于Transformer架构的条件序列建模在处理离线强化学习(RL)任务中已展现出其有效性,但在处理分布外状态和动作时仍存在困难。现有工作试图通过使用学习到的策略进行数据增强或使用基于值的RL算法添加额外约束来解决此问题。然而,这些研究仍未能克服以下挑战:(1)未能充分利用步骤间的历史时间信息,(2)忽视了返回目标(RTGs)、状态和动作之间的局部步骤内关系,(3)对带有噪声标签的次优轨迹过拟合。为解决这些挑战,我们提出了决策Mamba(DM),一种新颖的多粒度状态空间模型(SSM),并采用自进化策略学习策略。DM通过使用Mamba架构显式建模历史隐藏状态以提取时间信息。为了捕获RTG-状态-动作三元组之间的关系,设计了一个细粒度SSM模块并将其集成到Mamba中原始的粗粒度SSM中,从而产生一种专为离线RL定制的新型Mamba架构。最后,为缓解对噪声轨迹的过拟合问题,提出了一种使用渐进正则化的自进化策略。该策略利用自身过去的知识来细化次优动作,从而增强其对噪声演示的鲁棒性。在各种任务上的大量实验表明,DM显著优于其他基线方法。

英文摘要:

While the conditional sequence modeling with the transformer architecture has demonstrated its effectiveness in dealing with offline reinforcement learning (RL) tasks, it is struggle to handle out-of-distribution states and actions. Existing work attempts to address this issue by data augmentation with the learned policy or adding extra constraints with the value-based RL algorithm. However, these studies still fail to overcome the following challenges: (1) insufficiently utilizing the historical temporal information among inter-steps, (2) overlooking the local intrastep relationships among return-to-gos (RTGs), states, and actions, (3) overfitting suboptimal trajectories with noisy labels. To address these challenges, we propose Decision Mamba (DM), a novel multi-grained state space model (SSM) with a self-evolving policy learning strategy. DM explicitly models the historical hidden state to extract the temporal information by using the mamba architecture. To capture the relationship among RTG-state-action triplets, a fine-grained SSM module is designed and integrated into the original coarse-grained SSM in mamba, resulting in a novel mamba architecture tailored for offline RL. Finally, to mitigate the overfitting issue on noisy trajectories, a self-evolving policy is proposed by using progressive regularization. The policy evolves by using its own past knowledge to refine the suboptimal actions, thus enhancing its robustness on noisy demonstrations. Extensive experiments on various tasks show that DM outperforms other baselines substantially.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑