评论家引导的决策Transformer用于离线强化学习
Critic-Guided Decision Transformer for Offline Reinforcement Learning
AI总结：
提出评论家引导的决策Transformer（CGDT），融合值函数与决策Transformer，解决RCSL中采样与期望返回不一致问题，在随机环境和D4RL上超越传统方法。
摘要：
近期离线强化学习（RL）的进展凸显了返回条件监督学习（RCSL）的能力，该范式以监督方式基于每个状态的目标返回学习动作分布。然而，现有的RCSL方法主要关注确定性轨迹建模，忽视了随机状态转移和未来轨迹分布的多样性。一个根本性挑战源于单个轨迹内采样返回与跨多个轨迹的期望返回之间的不一致性。幸运的是，基于值的方法通过利用值函数近似期望返回提供了解决方案，从而有效解决了这一不一致性。基于这些见解，我们提出了一种新方法，称为评论家引导的决策Transformer（CGDT），它结合了基于值方法的长期返回可预测性与决策Transformer的轨迹建模能力。通过引入一个学习到的值函数（称为评论家），CGDT确保了指定目标返回与动作期望返回之间的直接对齐。这种集成弥合了RCSL的确定性与基于值方法的概率特性之间的差距。在随机环境和D4RL基准数据集上的实证评估表明，CGDT优于传统RCSL方法。这些结果突显了CGDT在推进离线RL最先进水平及将RCSL应用扩展到广泛RL任务方面的潜力。
英文摘要：
Recent advancements in offline reinforcement learning (RL) have underscored the capabilities of Return-Conditioned Supervised Learning (RCSL), a paradigm that learns the action distribution based on target returns for each state in a supervised manner. However, prevailing RCSL methods largely focus on deterministic trajectory modeling, disregarding stochastic state transitions and the diversity of future trajectory distributions. A fundamental challenge arises from the inconsistency between the sampled returns within individual trajectories and the expected returns across multiple trajectories. Fortunately, value-based methods offer a solution by leveraging a value function to approximate the expected returns, thereby addressing the inconsistency effectively. Building upon these insights, we propose a novel approach, termed the Critic-Guided Decision Transformer (CGDT), which combines the predictability of long-term returns from value-based methods with the trajectory modeling capability of the Decision Transformer. By incorporating a learned value function, known as the critic, CGDT ensures a direct alignment between the specified target returns and the expected returns of actions. This integration bridges the gap between the deterministic nature of RCSL and the probabilistic characteristics of value-based methods. Empirical evaluations on stochastic environments and D4RL benchmark datasets demonstrate the superiority of CGDT over traditional RCSL methods. These results highlight the potential of CGDT to advance the state of the art in offline RL and extend the applicability of RCSL to a wide range of RL tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
