arXiv 2409.080622024-09-12cs.LGcs.RO

面向离线强化学习的Q值正则化Decision ConvFormer

Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning

College of Applied Sciences, Shenzhen University（深圳大学应用学院）；Sino-German College of Intelligent Manufacturing, Shenzhen Technology University（深圳技术大学中德智能制造学院）

Teng Yan, Zhendong Ruan, Yaobang Cai, Yu Han, Wenxian Li, Yang Zhang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对离线强化学习中Decision Transformer存在的采样回报与最优回报不一致、轨迹拼接难的问题，提出结合动态规划动作值最大化项的Q值正则化Decision ConvFormer，在D4RL基准上性能优异，轨迹拼接能力突出。

摘要：

作为一种数据驱动范式，离线强化学习（Offline RL）已被建模为序列建模问题，其中Decision Transformer（DT，决策Transformer）展现出了卓越的性能。与以往拟合价值函数或计算策略梯度的强化学习方法不同，DT基于期望回报、历史状态和动作调整自回归模型，使用带因果掩码的Transformer输出最优动作。然而，由于单条轨迹内的采样回报与多条轨迹间的最优回报存在不一致性，设置期望回报以输出最优动作并拼接次优轨迹颇具挑战。Decision ConvFormer（DC，决策卷积Transformer）在马尔可夫决策过程下的强化学习轨迹建模语境中，比DT更易理解。我们提出Q值正则化Decision ConvFormer（QDC），该方法结合了DC对强化学习轨迹的理解，并在训练过程中引入了一项利用动态规划方法最大化动作值的项，确保采样动作的期望回报与最优回报保持一致。QDC在D4RL基准上取得了优异性能，在所有测试环境中均优于或接近最优水平，尤其在轨迹拼接能力上展现出突出的竞争力。

英文摘要：

As a data-driven paradigm, offline reinforcement learning (Offline RL) has been formulated as sequence modeling, where the Decision Transformer (DT) has demonstrated exceptional capabilities. Unlike previous reinforcement learning methods that fit value functions or compute policy gradients, DT adjusts the autoregressive model based on the expected returns, past states, and actions, using a causally masked Transformer to output the optimal action. However, due to the inconsistency between the sampled returns within a single trajectory and the optimal returns across multiple trajectories, it is challenging to set an expected return to output the optimal action and stitch together suboptimal trajectories. Decision ConvFormer (DC) is easier to understand in the context of modeling RL trajectories within a Markov Decision Process compared to DT. We propose the Q-value Regularized Decision ConvFormer (QDC), which combines the understanding of RL trajectories by DC and incorporates a term that maximizes action values using dynamic programming methods during training. This ensures that the expected returns of the sampled actions are consistent with the optimal returns. QDC achieves excellent performance on the D4RL benchmark, outperforming or approaching the optimal level in all tested environments. It particularly demonstrates outstanding competitiveness in trajectory stitching capability.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。