Q值正则化Transformer用于离线强化学习
Q-value Regularized Transformer for Offline Reinforcement Learning
AI总结：
针对离线强化学习中条件序列建模难以拼接最优轨迹的问题，提出Q值正则化Transformer（QT），结合Transformer轨迹建模与动态规划价值预测，在D4RL基准上超越传统方法。
摘要：
离线强化学习（RL）的最新进展凸显了条件序列建模（CSM）的能力，该范式基于历史轨迹和目标回报学习每个状态的动作分布。然而，这些方法常常因单个轨迹内的采样回报与跨多条轨迹的最优回报之间的不一致，而难以从次优轨迹中拼接出最优轨迹。幸运的是，动态规划（DP）方法通过利用价值函数近似每个状态的最优未来回报提供了一种解决方案，但这些技术容易产生不稳定的学习行为，特别是在长时域和稀疏奖励场景中。基于这些见解，我们提出了Q值正则化Transformer（QT），它结合了Transformer的轨迹建模能力与DP方法对最优未来回报的预测能力。QT学习一个动作价值函数，并将最大化动作值的项整合到CSM的训练损失中，旨在寻找与行为策略紧密对齐的最优动作。在D4RL基准数据集上的实证评估表明，QT优于传统的DP和CSM方法，凸显了QT提升离线RL最先进水平的潜力。
英文摘要：
Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Conditional Sequence Modeling (CSM), a paradigm that learns the action distribution based on history trajectory and target returns for each state. However, these methods often struggle with stitching together optimal trajectories from sub-optimal ones due to the inconsistency between the sampled returns within individual trajectories and the optimal returns across multiple trajectories. Fortunately, Dynamic Programming (DP) methods offer a solution by leveraging a value function to approximate optimal future returns for each state, while these techniques are prone to unstable learning behaviors, particularly in long-horizon and sparse-reward scenarios. Building upon these insights, we propose the Q-value regularized Transformer (QT), which combines the trajectory modeling ability of the Transformer with the predictability of optimal future returns from DP methods. QT learns an action-value function and integrates a term maximizing action-values into the training loss of CSM, which aims to seek optimal actions that align closely with the behavior policy. Empirical evaluations on D4RL benchmark datasets demonstrate the superiority of QT over traditional DP and CSM methods, highlighting the potential of QT to enhance the state-of-the-art in offline RL.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
