We study diffusion-based world models for reinforcement learning, which offer high generative fidelity but face critical efficiency challenges in control. Current methods either require heavyweight models at inference or rely on highly sequential imagination, both of which impose prohibitive computational costs. We propose Horizon Imagination (HI), an on-policy imagination process for discrete stochastic policies that denoises multiple future observations in parallel. HI incorporates a stabilization mechanism and a novel sampling schedule that decouples the denoising budget from the effective horizon over which denoising is applied while also supporting sub-frame budgets. Experiments on Atari 100K and Craftium show that our approach maintains control performance with a sub-frame budget of half the denoising steps and achieves superior generation quality under varied schedules. Code is available at https://github.com/leor-c/horizon-imagination.
Trust-Calibrated VLA Policy Post-Training with Evolving Imagination
Sun Yat-sen University（中山大学）；Shenzhen Loop Area Institute（深圳河套学院）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai AI Laboratory（上海人工智能实验室）；University of Oxford（牛津大学）；University of Science and Technology of China（中国科学技术大学）
GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control
Department of Computer Science and Engineering, Visvesvaraya Technological University (VTU), Belagavi, India（维斯瓦拉亚科技大学计算机科学与工程系，贝拉加维，印度）；Aliens on Earth (AoE) Autonomous Research Group, Belagavi, India（地球外星人自主研究组，贝拉加维，印度）
Open-World Reinforcement Learning over Long Short-Term Imagination
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能大模型关键实验室、人工智能研究院、上海交通大学）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究院、技术东院）；School of Computer Science and Technology, East China Normal University（计算机科学与技术学院、华东师范大学）
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学）；Peking University（北京大学）
The learned policy of model-free offline reinforcement learning (RL) methods is often constrained to stay within the support of datasets to avoid possible dangerous out-of-distribution actions or states, making it challenging to handle out-of-support region. Model-based RL methods offer a richer dataset and benefit generalization by generating imaginary trajectories with either trained forward or reverse dynamics model. However, the imagined transitions may be inaccurate, thus downgrading the performance of the underlying offline RL method. In this paper, we propose to augment the offline data…
In offline reinforcement learning (offline RL), one of the main challenges is to deal with the distributional shift between the learning policy and the given dataset. To address this problem, recent offline RL methods attempt to introduce conservatism bias to encourage learning in high-confidence areas. Model-free approaches directly encode such bias into policy or value function learning using conservative regularizations or special network structures, but their constrained policy search limits the generalization beyond the offline dataset. Model-based approaches learn forward dynamics models…
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Beihang University（北京航空航天大学）；Zhongguancun Academy（中关村学院）；Communication University of China（中国传媒大学）；Peking University（北京大学）；Hangzhou Innovation Institute of Beihang University（北京航空航天大学杭州创新研究院）
The University of Tokyo（东京大学）；Sony AI（索尼人工智能研究所）；Sony CSL（索尼计算机科学实验室）
Multi-agent RL is rendered difficult due to the non-stationary nature of environment perceived by individual agents. Theoretically sound methods using the REINFORCE estimator are impeded by its high-variance, whereas value-function based methods are affected by issues stemming from their ad-hoc handling of situations like inter-agent communication. Methods like MADDPG are further constrained due to their requirement of centralized critics etc. In order to address these issues, we present MA-Dreamer, a model-based method that uses both agent-centric and global differentiable models of the envir…
Reinforcement learning has seen great advancements in the past five years. The successful introduction of deep learning in place of more traditional methods allowed reinforcement learning to scale to very complex domains achieving super-human performance in environments like the game of Go or numerous video games. Despite great successes in multiple domains, these new methods suffer from their own issues that make them often inapplicable to the real world problems. Extreme lack of data efficiency, together with huge variance and difficulty in enforcing safety constraints, is one of the three m…
University of Toronto（多伦多大学）；Google Brain（谷歌大脑）；DeepMind（深度思维）
Learned world models summarize an agent's experience to facilitate learning complex behaviors. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks from images purely by latent imagination. We efficiently learn behaviors by propagating analytic gradients of learned state values back through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual contro…
Assuming distributions are Gaussian often facilitates computations that are otherwise intractable. We study the performance of an agent that attains a bounded information ratio with respect to a bandit environment with a Gaussian prior distribution and a Gaussian likelihood function when applied instead to a Bernoulli bandit. Relative to an information-theoretic bound on the Bayesian regret the agent would incur when interacting with the Gaussian bandit, we bound the increase in regret when the agent interacts with the Bernoulli bandit. If the Gaussian prior distribution and likelihood functio…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
Existing model-based value expansion methods typically leverage a world model for value estimation with a fixed rollout horizon to assist policy learning. However, the fixed rollout with an inaccurate model has a potential to harm the learning process. In this paper, we investigate the idea of using the model knowledge for value expansion adaptively. We propose a novel method called Dynamic-horizon Model-based Value Expansion (DMVE) to adjust the world model usage with different rollout horizons. Inspired by reconstruction-based techniques that can be applied for visual data novelty detection,…
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Duke University（杜克大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）
Institute of Science and Technology Austria（奥地利科学技术研究所）；Massachusetts Institute of Technology（麻省理工学院）；Technische Universität Wien（维也纳工业大学）
World models learn behaviors in a latent imagination space to enhance the sample-efficiency of deep reinforcement learning (RL) algorithms. While learning world models for high-dimensional observations (e.g., pixel inputs) has become practicable on standard RL benchmarks and some games, their effectiveness in real-world robotics applications has not been explored. In this paper, we investigate how such agents generalize to real-world autonomous vehicle control tasks, where advanced model-free deep RL algorithms fail. In particular, we set up a series of time-lap tasks for an F1TENTH racing rob…
When Denser Credit Is Not Enough: Evidence-Calibrated Policy Optimization for Long-Horizon LLM Agent Training
X-LANCE Lab, School of Computer Science, Shanghai Jiao Tong University, Shanghai, China（X-LANCE实验室，计算机科学学院，上海交通大学，上海，中国）；Faculty of Electronic and Information Engineering, Xi’an Jiaotong University（电子与信息工程学院，西安交通大学）
We consider finite and infinite horizon dynamic programming problems, where the control at each stage consists of several distinct decisions, each one made by one of several agents. We introduce an approach, whereby at every stage, each agent's decision is made by executing a local rollout algorithm that uses a base policy, together with some coordinating information from the other agents. The amount of local computation required at every stage by each agent is independent of the number of agents, while the amount of total computation (over all agents) grows linearly with the number of agents.…
Mind Dreamer: Untethering Imagination via Active Causal Intervention on Latent Manifolds
Center for Brain-Inspired Computing Research, Department of Precision Instrument, Tsinghua University, Beijing, China（脑启发计算研究中心，精密仪器系，清华大学，北京，中国）；College of Computer Science and Technology, Zhejiang University, Hangzhou, China（计算机科学与技术学院，浙江大学，杭州，中国）；Pen-Tung Sah Institute of Micro-Nano Science and Technology, Xiamen University, Xiamen, China（彭途萨微纳米科学与技术研究院，厦门大学，厦门，中国）
Koopman Dreamer: Spectrally Constrained Latent Dynamics for Stable World-Model Imagination
College of Intelligence Science and Technology, National University of Defense Technology（国防科技大学智能科学与技术学院）；School of Engineering, Newcastle University（纽卡斯尔大学工程学院）
Model usage is the central challenge of model-based reinforcement learning. Although dynamics model based on deep neural networks provide good generalization for single step prediction, such ability is over exploited when it is used to predict long horizon trajectories due to compounding errors. In this work, we propose a Dyna-style model-based reinforcement learning algorithm, which we called Maximum Entropy Model Rollouts (MEMR). To eliminate the compounding errors, we only use our model to generate single-step rollouts. Furthermore, we propose to generate \emph{diverse} model rollouts by no…
Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Huawei（华为）
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；College of Information and Computer Sciences（信息与计算机科学学院）
Model-based reinforcement learning promises to learn an optimal policy from fewer interactions with the environment compared to model-free reinforcement learning by learning an intermediate model of the environment in order to predict future interactions. When predicting a sequence of interactions, the rollout length, which limits the prediction horizon, is a critical hyperparameter as accuracy of the predictions diminishes in the regions that are further away from real experience. As a result, with a longer rollout length, an overall worse policy is learned in the long run. Thus, the hyperpar…
Restless multi-armed bandits with partially observable states has applications in communication systems, age of information and recommendation systems. In this paper, we study multi-state partially observable restless bandit models. We consider three different models based on information observable to decision maker -- 1) no information is observable from actions of a bandit 2) perfect information from bandit is observable only for one action on bandit, there is a fixed restart state, i.e., transition occurs from all other states to that state 3) perfect state information is available to decis…
Learning with a Single Rollout via Pass@k Policy Optimization
University of Alberta（阿尔伯塔大学）；BIGAI（北京通用人工智能研究院）；University of Toronto（多伦多大学）；McGill University, Mila（麦吉尔大学，米拉）；Nvidia Research（英伟达研究院）；Amii（阿尔伯塔机器智能研究所）；CIFAR AI Chair（CIFAR人工智能教席）
Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
School of Information, Renmin University of China（中国人民大学信息学院）；Key Laboratory of Data Engineering and Knowledge Engineering（数据工程与知识工程重点实验室）；Ant Group（蚂蚁集团）
Florida Institute for Human and Machine Cognition (IHMC)（佛罗里达人机认知研究所（IHMC））
Despite advancements in deep reinforcement learning algorithms, developing an effective exploration strategy is still an open problem. Most existing exploration strategies either are based on simple heuristics, or require the model of the environment, or train additional deep neural networks to generate imagination-augmented paths. In this paper, a revolutionary algorithm, called Policy Augmentation, is introduced. Policy Augmentation is based on a newly developed inductive matrix completion method. The proposed algorithm augments the values of unexplored state-action pairs, helping the agent …
SearchWorld: Spatial Value-Grounded Imagination for UAV Object Search via World Models
National Key Laboratory of Digital Intelligent Modeling and Simulation, National University of Defense Technology（国防科技大学数字智能建模与仿真全国重点实验室）；College of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院）；Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院空天信息创新研究院）；BNRist, Tsinghua University（清华大学北京国家研究中心）；Southeast University（东南大学）
Lucid Dreaming for World Models: Learning to Doubt Imagination and Decide by Trust
National University of Singapore（新加坡国立大学）；King Abdullah University of Science and Technology (KAUST)（阿卜杜拉国王科技大学）；Agency for Science, Technology and Research (A*STAR)（新加坡科技研究局）
HERO: History-Enriched Rollout Training for Long-Horizon Autoregressive Neural Operators
School of Computer Science and Engineering, UESTC（电子科技大学计算机科学与工程学院）；Electronics and Information Convergence Engineering, KHU（明知大学电子信息融合工程系）；Department of Mathematical Sciences, UOL（利物浦大学数学科学系）；School of Computer Science and Technology, XDU（西安电子科技大学计算机科学与技术学院）；School of Mechanical and Electrical Engineering, UESTC（电子科技大学机械与电气工程学院）；College of Computer and Information Engineering, XAUAT（西安建筑科技大学计算机与信息工程学院）
