提出Diffusion World Model（DWM），通过单次前向传播同时预测多步未来状态与奖励，集成于离线强化学习的值估计，在D4RL上较一步模型提升44%性能。
摘要：
我们引入了Diffusion World Model（DWM），这是一种能够同时预测多步未来状态和奖励的条件扩散模型。与传统的一步动力学模型不同，DWM在单次前向传播中提供长程预测，消除了递归查询的需要。我们将DWM集成到基于模型的值估计中，其中短期回报由从DWM采样的未来轨迹模拟。在离线强化学习背景下，DWM可以被视为通过生成建模实现的保守值正则化。或者，它也可以被看作是使带有合成数据的离线Q-learning成为可能的数据源。我们在D4RL数据集上的实验证实了DWM对长程模拟的鲁棒性。在绝对性能方面，DWM以44%的性能提升显著超越了一步动力学模型，并且与无模型对应方法相当或略胜一筹。
英文摘要：
We introduce Diffusion World Model (DWM), a conditional diffusion model capable of predicting multistep future states and rewards concurrently. As opposed to traditional one-step dynamics models, DWM offers long-horizon predictions in a single forward pass, eliminating the need for recursive queries. We integrate DWM into model-based value estimation, where the short-term return is simulated by future trajectories sampled from DWM. In the context of offline reinforcement learning, DWM can be viewed as a conservative value regularization through generative modeling. Alternatively, it can be seen as a data source that enables offline Q-learning with synthetic data. Our experiments on the D4RL dataset confirm the robustness of DWM to long-horizon simulation. In terms of absolute performance, DWM significantly surpasses one-step dynamics models with a $44\%$ performance gain, and is comparable to or slightly surpassing their model-free counterparts.
How Much Online RL is Enough? Informative Rollouts for Offline Preference Optimization in RLVR
TCS Research Department of CSE（TCS计算机科学系研究部）；IIT Madras（印度理工学院马德拉斯分校）；Department of Data Science & AI（数据科学与人工智能系）；Wadhwani School of Data Science & AI（Wadhwani数据科学与人工智能学院）
SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanyang Technological University（南洋理工大学）
Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Huawei（华为）
Environment Transformer and Policy Optimization for Model-Based Offline Reinforcement Learning
The Hong Kong University of Science and Technology（香港科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangdong Provincial Key Lab of Integrated Communication, Sensing and Computation for Ubiquitous Internet of Things（广东省泛在物联网融合通信感知计算重点实验室）
Model-Based Offline Reinforcement Learning with Adversarial Data Augmentation
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学现代软件技术国家重点实验室）；City University of Hong Kong（香港城市大学）；School of Computer Science, Nanjing University of Posts and Telecommunications（南京邮电大学计算机学院）；Department of Computer Science, University of Liverpool（利物浦大学计算机系）
EchoRL: Reinforcement Learning via Rollout Echoing
Munich Center for Machine Learning（慕尼黑机器学习中心）；Huawei Heisenberg Research Center（华为海森堡研究所以）；University of Arizona（亚利桑那大学）；College of Computing（计算学院）；Data Science, Nanyang Technological University, Singapore（数据科学，南洋理工大学，新加坡）；MemAgents Lab（MemAgents实验室）
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Carnegie Mellon University（卡内基梅隆大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Meta AI
RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
Shanghai Jiao Tong University（上海交通大学）；Central South University（中南大学）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Beihang University（北京航空航天大学）
DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
School of Computer Science \& Beijing Key Laboratory of Software；Hardware Cooperative Artificial Intelligence Systems, Peking University, Beijing, China；School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China；Institute of Computational Social Science, Peking University (Qingdao), Qingdao, China
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算与智能学院）；Beijing Zhongguancun Academy（北京中关村学院）；XinzhuAI（芯智科技）
APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
Advanced Micro Devices, Inc. (AMD)（先进微器件公司）；Carnegie Mellon University (CMU)（卡内基梅隆大学）；LMSYS Org（LMSYS组织）；University of California, Los Angeles (UCLA)（加州大学洛杉矶分校）
Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations
Georgia Institute of Technology（佐治亚理工学院）；Amazon Alexa AI（亚马逊Alexa人工智能）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）；University of Southern California（南加州大学）
SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache
Cornell University（康奈尔大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Tsinghua University（清华大学）；University of Washington（华盛顿大学）；ByteDance（字节跳动）
Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models
Seoul National University（首尔大学）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔大学人工智能交叉项目）；Artificial Intelligence Institute, Seoul National University（首尔大学人工智能研究院）；Department of Intelligence and Information, Seoul National University（首尔大学智能与信息系）
