Inspired by the success of the Transformer architecture in natural language processing and computer vision, we investigate the use of Transformers in Reinforcement Learning (RL), specifically in modeling the environment's dynamics using Transformer Dynamics Models (TDMs). We evaluate the capabilities of TDMs for continuous control in real-time planning scenarios with Model Predictive Control (MPC). While Transformers excel in long-horizon prediction, their tokenization mechanism and autoregressive nature lead to costly planning over long horizons, especially as the environment's dimensionality increases. To alleviate this issue, we use a TDM for short-term planning, and learn an autoregressive discrete Q-function using a separate Q-Transformer (QT) model to estimate a long-term return beyond the short-horizon planning. Our proposed method, QT-TDM, integrates the robust predictive capabilities of Transformers as dynamics models with the efficacy of a model-free Q-Transformer to mitigate the computational burden associated with real-time planning. Experiments in diverse state-based continuous control tasks show that QT-TDM is superior in performance and sample efficiency compared to existing Transformer-based RL models while achieving fast and computationally efficient inference.
Planning in Dynamic Environments with Conditional Autoregressive Models
我们演示了条件自回归生成模型的使用(van den Oord等人,2016 a)在离散潜在空间上(van den Oord等人,2017年b)与MCTS进行前瞻性规划。为了测试这种方法,我们引入了一个新的环境,具有不同的难度水平,以及移动的目标和障碍。高质量的帧生成和经典的规划方法的组合几乎匹配我们的任务的真实环境性能,证明了这种方法在动态环境中基于模型的规划的有用性。
Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspective
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Beijing Key Laboratory of Research on Large Models and Intelligent Governance(北京大模型与智能治理研究重点实验室);Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE(下一代智能搜索与推荐工程技术研究中心);Stanford University(斯坦福大学);Lambda, Inc(Lambda公司);Tsinghua University(清华大学)