MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning
Xiamen University（厦门大学）；Hong Kong University of Science and Technology（香港科技大学）；National University of Defense Technology（国防科技大学）；Jimei University（集美大学）
Generative models have gained significant traction in offline reinforcement learning (RL) due to their ability to model complex trajectory distributions. However, existing generation-based approaches still struggle with long-horizon tasks characterized by sparse rewards. Some hierarchical generation methods have been developed to mitigate this issue by decomposing the original problem into shorter-horizon subproblems using one policy and generating detailed actions with another. While effective, these methods often overlook the multi-scale temporal structure inherent in trajectories, resulting in suboptimal performance. To overcome these limitations, we propose MAGE, a Multi-scale Autoregressive GEneration-based offline RL method. MAGE incorporates a condition-guided multi-scale autoencoder to learn hierarchical trajectory representations, along with a multi-scale transformer that autoregressively generates trajectory representations from coarse to fine temporal scales. MAGE effectively captures temporal dependencies of trajectories at multiple resolutions. Additionally, a condition-guided decoder is employed to exert precise control over short-term behaviors. Extensive experiments on five offline RL benchmarks against fifteen baseline algorithms show that MAGE successfully integrates multi-scale trajectory modeling with conditional guidance, generating coherent and controllable trajectories in long-horizon sparse-reward settings.
University of Toronto（多伦多大学）；DeepMind；Google Brain（谷歌大脑）
Standard dynamics models for continuous control make use of feedforward computation to predict the conditional distribution of next state and reward given current state and action using a multivariate Gaussian with a diagonal covariance structure. This modeling choice assumes that different dimensions of the next state and reward are conditionally independent given the current state and action and may be driven by the fact that fully observable physics-based simulation environments entail deterministic transition dynamics. In this paper, we challenge this conditional independence assumption an…
Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL
Department of Electrical and Computer Engineering, Seoul National University, Seoul, South Korea（首尔国立大学电气与计算机工程系）；Department of Automotive Engineering, Ajou University, Gyeonggi-do, South Korea（全州大学汽车工程系）
In-Context Reinforcement Learning From Suboptimal Historical Data
Department of Electrical and Computer Engineering, Duke University, Durham, US（电气与计算机工程系，杜克大学，达勒姆，美国）；Department of Biostatistics and Bioinformatics, Duke University, Durham, US（生物统计学与生物信息学系，杜克大学，达勒姆，美国）；Department of Statistics and Data Science, Yale University, New Haven, US（统计学与数据科学系，耶鲁大学，新 Haven，美国）
Autoregressive generative models can estimate complex continuous data distributions, like trajectory rollouts in an RL environment, image intensities, and audio. Most state-of-the-art models discretize continuous data into several bins and use categorical distributions over the bins to approximate the continuous data distribution. The advantage is that the categorical distribution can easily express multiple modes and are straightforward to optimize. However, such approximation cannot express sharp changes in density without using significantly more bins, making it parameter inefficient. We pr…
Meta AI Research（Meta人工智能研究院）；University of California, Berkeley（加州大学伯克利分校）；University of California, Los Angeles（加州大学洛杉矶分校）
Recent work has shown that offline reinforcement learning (RL) can be formulated as a sequence modeling problem (Chen et al., 2021; Janner et al., 2021) and solved via approaches similar to large-scale language modeling. However, any practical instantiation of RL also involves an online component, where policies pretrained on passive offline datasets are finetuned via taskspecific interactions with the environment. We propose Online Decision Transformers (ODT), an RL algorithm based on sequence modeling that blends offline pretraining with online finetuning in a unified framework. Our framewor…
Q-value Regularized Decision ConvFormer for Offline Reinforcement Learning
College of Applied Sciences, Shenzhen University（深圳大学应用学院）；Sino-German College of Intelligent Manufacturing, Shenzhen Technology University（深圳技术大学中德智能制造学院）
Decoupling KL and Trajectories: A Unified Perspective for SFT, DAgger, Offline RL, and OPD in LLM Distillation
Eastern Institute of Technology（东技术院）；The Hong Kong Polytechnic University（香港理工大学）；Shanghai Jiao Tong University（上海交通大学）；Thrust of Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou)（人工智能 thrust，香港科学与技术大学（广州））
UC Berkeley（加州大学伯克利分校）；Facebook AI Research（脸书人工智能研究院）；Google Brain（谷歌大脑）
We introduce a framework that abstracts Reinforcement Learning (RL) as a sequence modeling problem. This allows us to draw upon the simplicity and scalability of the Transformer architecture, and associated advances in language modeling such as GPT-x and BERT. In particular, we present Decision Transformer, an architecture that casts the problem of RL as conditional sequence modeling. Unlike prior approaches to RL that fit value functions or compute policy gradients, Decision Transformer simply outputs the optimal actions by leveraging a causally masked Transformer. By conditioning an autoregr…
Courant Institute of Mathematical Sciences, New York University（纽约大学柯朗数学科学研究所）；Center for Data Science, New York University（纽约大学数据科学中心）
Current approaches to text generation largely rely on autoregressive models and maximum likelihood estimation. This paradigm leads to (i) diverse but low-quality samples due to mismatched learning objective and evaluation metric (likelihood vs. quality) and (ii) exposure bias due to mismatched history distributions (gold vs. model-generated). To alleviate these problems, we frame text generation as an offline reinforcement learning (RL) problem with expert demonstrations (i.e., the reference), where the goal is to maximize quality given model-generated histories. We propose GOLD (generation by…
Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models
