Current image-based reinforcement learning (RL) algorithms typically operate on the whole image without performing object-level reasoning. This leads to inefficient goal sampling and ineffective reward functions. In this paper, we improve upon previous visual self-supervised RL by incorporating object-level reasoning and occlusion reasoning. Specifically, we use unknown object segmentation to ignore distractors in the scene for better reward computation and goal generation; we further enable occlusion reasoning by employing a novel auxiliary loss and training scheme. We demonstrate that our proposed algorithm, ROLL (Reinforcement learning with Object Level Learning), learns dramatically faster and achieves better final performance compared with previous methods in several simulated visual control tasks. Project video and code are available at https://sites.google.com/andrew.cmu.edu/roll.
The Hong Kong University of Science and Technology（香港科技大学）
Traditional model-based reinforcement learning (RL) methods generate forward rollout traces using the learnt dynamics model to reduce interactions with the real environment. The recent model-based RL method considers the way to learn a backward model that specifies the conditional probability of the previous state given the previous action and the current state to additionally generate backward rollout trajectories. However, in this type of model-based method, the samples derived from backward rollouts and those from forward rollouts are simply aggregated together to optimize the policy via th…
the University of Manchester（曼彻斯特大学）；AIST Artificial Intelligence Research Centre（产业技术综合研究所人工智能研究中心）
Deep reinforcement learning has proven to be a great success in allowing agents to learn complex tasks. However, its application to actual robots can be prohibitively expensive. Furthermore, the unpredictability of human behavior in human-robot interaction tasks can hinder convergence to a good policy. In this paper, we present an architecture that allows agents to learn models of stochastic environments and use them to accelerate learning. We descirbe how an environment model can be learned online and used to generate synthetic transitions, as well as how an agent can leverage these synthetic…
SRT: Accelerating Reinforcement Learning via Speculative Rollout with Tree-Structured Cache
Cornell University（康奈尔大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Tsinghua University（清华大学）；University of Washington（华盛顿大学）；ByteDance（字节跳动）
Beyond On-Policy Exploration: Integrating External Policy Rollouts for Reinforcement Learning in Diffusion Language Models
Seoul National University（首尔大学）；Interdisciplinary Program in Artificial Intelligence, Seoul National University（首尔大学人工智能交叉项目）；Artificial Intelligence Institute, Seoul National University（首尔大学人工智能研究院）；Department of Intelligence and Information, Seoul National University（首尔大学智能与信息系）
SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learning
Harbin Institute of Technology, Shenzhen（哈尔滨工业大学（深圳））；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Nanyang Technological University（南洋理工大学）
Contextual Rollout Bandits for Reinforcement Learning with Verifiable Rewards
School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）；Huawei（华为）
RollVerify: Bridging Efficiency and Accuracy in Long-Tail Rollout Reinforcement Learning
Shanghai Jiao Tong University（上海交通大学）；Central South University（中南大学）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）；Beihang University（北京航空航天大学）
In recent years, Reinforcement Learning (RL) has seen increasing popularity in research and popular culture. However, skepticism still surrounds the practicality of RL in modern video game development. In this paper, we demonstrate by example that RL can be a great tool for Artificial Intelligence (AI) design in modern, non-trivial video games. We present our RL system for Ubisoft's Roller Champions, a 3v3 Competitive Multiplayer Sports Game played on an oval-shaped skating arena. Our system is designed to keep up with agile, fast-paced development, taking 1--4 days to train a new model follow…
University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）；College of Information and Computer Sciences（信息与计算机科学学院）
Model-based reinforcement learning promises to learn an optimal policy from fewer interactions with the environment compared to model-free reinforcement learning by learning an intermediate model of the environment in order to predict future interactions. When predicting a sequence of interactions, the rollout length, which limits the prediction horizon, is a critical hyperparameter as accuracy of the predictions diminishes in the regions that are further away from real experience. As a result, with a longer rollout length, an overall worse policy is learned in the long run. Thus, the hyperpar…
RolloutPipe: Overlapping Pipelined Rollout and Training in Disaggregated On-Policy LLM Reinforcement Learning
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Southern University of Science and Technology（南方科技大学）
Cross-Rollout Bellman Closure for Long-Horizon Agentic Reinforcement Learning
Beihang University（北京航空航天大学）；Zhongguancun Academy（中关村学院）；Communication University of China（中国传媒大学）；Peking University（北京大学）；Hangzhou Innovation Institute of Beihang University（北京航空航天大学杭州创新研究院）
ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rollouts
National Key Laboratory for Novel Software Technology Nanjing University China（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence Nanjing University China（南京大学人工智能学院）；The University of Hong Kong（香港大学）
EchoRL: Reinforcement Learning via Rollout Echoing
Munich Center for Machine Learning（慕尼黑机器学习中心）；Huawei Heisenberg Research Center（华为海森堡研究所以）；University of Arizona（亚利桑那大学）；College of Computing（计算学院）；Data Science, Nanyang Technological University, Singapore（数据科学，南洋理工大学，新加坡）；MemAgents Lab（MemAgents实验室）
GRPODropout: Less is More for Online Reinforcement Learning Rollouts
Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳）计算与智能学院）；Beijing Zhongguancun Academy（北京中关村学院）；XinzhuAI（芯智科技）
Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Carnegie Mellon University（卡内基梅隆大学）；Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Meta AI
ECHO-2: A Large-Scale Distributed Rollout Framework for Cost-Efficient Reinforcement Learning
The University of Hong Kong（香港大学）；Fudan University（复旦大学）；Gradient；University of Edinburgh（爱丁堡大学）；Soochow University（苏州大学）；Technical University of Darmstadt（达姆施塔特技术大学）；University of the Chinese Academy of Sciences（中国科学院大学）
We consider finite and infinite horizon dynamic programming problems, where the control at each stage consists of several distinct decisions, each one made by one of several agents. We introduce an approach, whereby at every stage, each agent's decision is made by executing a local rollout algorithm that uses a base policy, together with some coordinating information from the other agents. The amount of local computation required at every stage by each agent is independent of the number of agents, while the amount of total computation (over all agents) grows linearly with the number of agents.…
RASFT: Rollout-Adaptive Supervised Fine-Tuning for Reasoning
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；New Jersey Institute of Technology（新泽西理工学院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
There are two halves to RL systems: experience collection time and policy learning time. For a large number of samples in rollouts, experience collection time is the major bottleneck. Thus, it is necessary to speed up the rollout generation time with multi-process architecture support. Our work, dubbed WALL-E, utilizes multiple rollout samplers running in parallel to rapidly generate experience. Due to our parallel samplers, we experience not only faster convergence times, but also higher average reward thresholds. For example, on the MuJoCo HalfCheetah-v2 task, with $N = 10$ parallel sampler …
Sparse-RL: Breaking the Memory Wall in LLM Reinforcement Learning via Stable Sparse Rollouts
School of Information, Renmin University of China（中国人民大学信息学院）；Key Laboratory of Data Engineering and Knowledge Engineering（数据工程与知识工程重点实验室）；Ant Group（蚂蚁集团）
DARTS: Distribution-Aware Active Rollout Trajectory Shaping for Accelerating LLM Reinforcement Learning
School of Computer Science \& Beijing Key Laboratory of Software；Hardware Cooperative Artificial Intelligence Systems, Peking University, Beijing, China；School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China；Institute of Computational Social Science, Peking University (Qingdao), Qingdao, China
