arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.18841cs.LG
收藏

QT-TDM:基于Transformer动力学模型与自回归Q学习的规划

QT-TDM: Planning With Transformer Dynamics Model and Autoregressive Q-Learning

Universität Hamburg(汉堡大学);Aswan University(阿斯旺大学);University of Southampton(南安普顿大学)

Mostafa Kotb, Cornelius Weber, Muhammad Burhan Hafez, Stefan Wermter

AI总结:

本文提出QT-TDM方法,结合Transformer动力学模型进行短期规划与Q-Transformer进行长期回报估计,有效缓解了实时规划的计算负担,在连续控制任务中实现了优越的性能、样本效率和推理速度。

摘要:

受Transformer架构在自然语言处理和计算机视觉中取得成功的启发,我们研究了Transformer在强化学习(RL)中的应用,特别是使用Transformer动力学模型(TDM)对环境动力学进行建模。我们评估了TDM在模型预测控制(MPC)的实时规划场景中进行连续控制的能力。尽管Transformer在长期预测方面表现出色,但其分词机制和自回归特性导致在长周期上的规划成本高昂,尤其是随着环境维度的增加。为了缓解这一问题,我们使用TDM进行短期规划,并使用单独的Q-Transformer(QT)模型学习自回归离散Q函数,以估计超出短周期规划的长期回报。我们提出的方法QT-TDM,将Transformer作为动力学模型的强大预测能力与无模型Q-Transformer的有效性相结合,以减轻与实时规划相关的计算负担。在多样化的基于状态的连续控制任务中的实验表明,与现有的基于Transformer的RL模型相比,QT-TDM在性能和样本效率方面更优,同时实现了快速且计算高效的推理。

英文摘要:

Inspired by the success of the Transformer architecture in natural language processing and computer vision, we investigate the use of Transformers in Reinforcement Learning (RL), specifically in modeling the environment's dynamics using Transformer Dynamics Models (TDMs). We evaluate the capabilities of TDMs for continuous control in real-time planning scenarios with Model Predictive Control (MPC). While Transformers excel in long-horizon prediction, their tokenization mechanism and autoregressive nature lead to costly planning over long horizons, especially as the environment's dimensionality increases. To alleviate this issue, we use a TDM for short-term planning, and learn an autoregressive discrete Q-function using a separate Q-Transformer (QT) model to estimate a long-term return beyond the short-horizon planning. Our proposed method, QT-TDM, integrates the robust predictive capabilities of Transformers as dynamics models with the efficacy of a model-free Q-Transformer to mitigate the computational burden associated with real-time planning. Experiments in diverse state-based continuous control tasks show that QT-TDM is superior in performance and sample efficiency compared to existing Transformer-based RL models while achieving fast and computationally efficient inference.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑