Recently, incorporating knowledge from pretrained language models (PLMs) into decision transformers (DTs) has generated significant attention in offline reinforcement learning (RL). These PLMs perform well in RL tasks, raising an intriguing question: what kind of knowledge from PLMs has been transferred to RL to achieve such good results? This work first dives into this problem by analyzing each head quantitatively and points out Markov head, a crucial component that exists in the attention heads of PLMs. It leads to extreme attention on the last-input token and performs well only in short-term environments. Furthermore, we prove that this extreme attention cannot be changed by re-training embedding layer or fine-tuning. Inspired by our analysis, we propose a general method GPT2-DTMA, which equips a pretrained DT with Mixture of Attention (MoA), to accommodate diverse attention requirements during fine-tuning. Extensive experiments corroborate our theorems and demonstrate the effectiveness of GPT2-DTMA: it achieves comparable performance in short-term environments while significantly narrowing the performance gap in long-term environments.
Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Alibaba Group（阿里巴巴集团）
FOSP: Fine-tuning Offline Safe Policy through World Models
Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；College of Design and Engineering, National University of Singapore（新加坡国立大学设计与工程学院）
Meta AI Research（Meta人工智能研究院）；University of California, Berkeley（加州大学伯克利分校）；University of California, Los Angeles（加州大学洛杉矶分校）
最近的工作表明，离线强化学习（RL）可以被公式化为序列建模问题（Chen et al.，2021; Janner等人，2021年），并通过类似于大规模语言建模的方法解决。然而，RL的任何实际实例化都涉及在线组件，其中在被动离线数据集上预训练的策略通过与环境的特定任务交互进行微调。我们提出了在线决策Transformers（ODT），一种基于序列建模的RL算法，在一个统一的框架中将离线预训练与在线微调相结合。我们的框架结合自回归建模目标使用序列级熵正则化器，以进行样本有效的探索和微调。从经验上讲，我们表明，ODT是有竞争力的国家的最先进的绝对性能的D4 RL基准，但在微调过程中显示出更显着的收益。
Department of Automation, Tsinghua University（清华大学自动化系）；Beijing Institute of General Artificial Intelligence (BIGAI)（北京通用人工智能研究院）；Department of Statistics and Data Science, UCLA（UCLA统计与数据科学系）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Department of Computer Science, UCLA（UCLA计算机科学系）
A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
The University of Sydney（悉尼大学）；SenseTime Computer Vision Group（商汤科技计算机视觉组）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；CUHK MMLab（香港中文大学多媒体实验室）；Institute for AI, Peking University（北京大学人工智能研究院）
LEASE: Offline Preference-based Reinforcement Learning with High Sample Efficiency
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；CASIA-MUST Joint Laboratory of Intelligence Science and Technology, Institute of Systems Engineering, Macau University of Science and Technology（智能科学与技术CASIA-MUST联合实验室，澳门科学理工学院）
Inference Time Policy Optimization for Offline RL with Differentiable World Models
Siebel School of Computing and Data Science（计算与数据科学学院）；Department of Computer Sciences（计算机科学系）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Wisconsin Madison（威斯康星大学麦迪逊分校）
Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院教育部人工智能重点实验室）；Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China（东方理工高等研究院宁波数字孪生研究院）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Ningbo, China（宁波市空间智能与数字衍生重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Shenyang Institute of Computing Technology, Chinese Academy of Sciences（中国科学院沈阳计算技术研究所）；Shenyang CASNC Technology Co., Ltd（沈阳中科数控技术股份有限公司）
