We present ProgAgent, a continual reinforcement learning (CRL) agent that unifies progress-aware reward learning with a high-throughput, JAX-native system architecture. Lifelong robotic learning grapples with catastrophic forgetting and the high cost of reward specification. ProgAgent tackles these by deriving dense, shaped rewards from unlabeled expert videos through a perceptual model that estimates task progress across initial, current, and goal observations. We theoretically interpret this as a learned state-potential function, delivering robust guidance in line with expert behaviors. To maintain stability amid online exploration - where novel, out-of-distribution states arise - we incorporate an adversarial push-back refinement that regularizes the reward model, curbing overconfident predictions on non-expert trajectories and countering distribution shift. By embedding this reward mechanism into a JIT-compiled loop, ProgAgent supports massively parallel rollouts and fully differentiable updates, rendering a sophisticated unified objective feasible: it merges PPO with coreset replay and synaptic intelligence for an enhanced stability-plasticity balance. Evaluations on ContinualBench and Meta-World benchmarks highlight ProgAgent's advantages: it markedly reduces forgetting, boosts learning speed, and outperforms key baselines in visual reward learning (e.g., Rank2Reward, TCN) and continual learning (e.g., Coreset, SI) - surpassing even an idealized perfect memory agent. Real-robot trials further validate its ability to acquire complex manipulation skills from noisy, few-shot human demonstrations.
Institute of Intelligent Machines, Hefei Institutes of Physical Science, Chinese Academy of Sciences（中国科学院合肥物质科学研究院智能机械研究所）；University of Science and Technology of China（中国科学技术大学）
Exploration with sparse rewards remains a challenging research problem in reinforcement learning (RL). Especially for sequential object manipulation tasks, the RL agent always receives negative rewards until completing all sub-tasks, which results in low exploration efficiency. To solve these tasks efficiently, we propose a novel self-guided continual RL framework, RelayHER (RHER). RHER first decomposes a sequential task into new sub-tasks with increasing complexity and ensures that the simplest sub-task can be learned quickly by utilizing Hindsight Experience Replay (HER). Secondly, we design…
University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所（Amii））
Learning auxiliary tasks, such as multiple predictions about the world, can provide many benefits to reinforcement learning systems. A variety of off-policy learning algorithms have been developed to learn such predictions, but as yet there is little work on how to adapt the behavior to gather useful data for those off-policy predictions. In this work, we investigate a reinforcement learning system designed to learn a collection of auxiliary tasks, with a behavior policy learning to take actions to improve those auxiliary predictions. We highlight the inherent non-stationarity in this continua…
Monash University（莫纳什大学）；South China University of Technology（华南理工大学）
Deep reinforcement learning has made significant progress in the field of continuous control, such as physical control and autonomous driving. However, it is challenging for a reinforcement model to learn a policy for each task sequentially due to catastrophic forgetting. Specifically, the model would forget knowledge it learned in the past when trained on a new task. We consider this challenge from two perspectives: i) acquiring task-specific skills is difficult since task information and rewards are not highly related; ii) learning knowledge from previous experience is difficult in continuou…
Berkeley AI Research（伯克利人工智能研究院）；Stanford（斯坦福大学）；Mila（米拉人工智能研究院）
We develop a new continual meta-learning method to address challenges in sequential multi-task learning. In this setting, the agent's goal is to achieve high reward over any sequence of tasks quickly. Prior meta-reinforcement learning algorithms have demonstrated promising results in accelerating the acquisition of new tasks. However, they require access to all tasks during training. Beyond simply transferring past experience to new tasks, our goal is to devise continual reinforcement learning algorithms that learn to learn, using their experience on previous tasks to learn new tasks more quic…
Non-asymptotic Convergence of Average-reward Q-learning with Options
Indian Institute of Science（印度科学学院）；Universitat Pompeu Fabra（庞培法布拉大学）；Univ. Lille, Inria, CNRS- CRIStAL, Centrale Lille（里尔大学、Inria、法国国家科学研究中心CRIStAL实验室、里尔中央理工学院）
Continual Learning (CL) considers the problem of training an agent sequentially on a set of tasks while seeking to retain performance on all previous tasks. A key challenge in CL is catastrophic forgetting, which arises when performance on a previously mastered task is reduced when learning a new task. While a variety of methods exist to combat forgetting, in some cases tasks are fundamentally incompatible with each other and thus cannot be learnt by a single policy. This can occur, in reinforcement learning (RL) when an agent may be rewarded for achieving different goals from the same observa…
Reward shaping allows reinforcement learning (RL) agents to accelerate learning by receiving additional reward signals. However, these signals can be difficult to design manually, especially for complex RL tasks. We propose a simple and general approach that determines the reward of pre-defined events by their rarity alone. Here events become less rewarding as they are experienced more often, which encourages the agent to continually explore new types of events as it learns. The adaptiveness of this reward function results in a form of automated curriculum learning that does not have to be spe…
A core aspect of human intelligence is the ability to learn new tasks quickly and switch between them flexibly. Here, we describe a modular continual reinforcement learning paradigm inspired by these abilities. We first introduce a visual interaction environment that allows many types of tasks to be unified in a single framework. We then describe a reward map prediction scheme that learns new tasks robustly in the very large state and action spaces required by such an environment. We investigate how properties of module architecture influence efficiency of task learning, showing that a module …
Future of Humanity Institute, Oxford University（牛津大学人类未来研究所）；Machine Intelligence Research Institute（机器智能研究所）；DeepMind
In some agent designs like inverse reinforcement learning an agent needs to learn its own reward function. Learning the reward function and optimising for it are typically two different processes, usually performed at different stages. We consider a continual (``one life'') learning approach where the agent both learns the reward function and optimises for it at the same time. We show that this comes with a number of pitfalls, such as deliberately manipulating the learning process in one direction, refusing to learn, ``learning'' facts already known to the agent, and making decisions that are …
Johannes Kepler University Linz（林茨约翰内斯·开普勒大学）；Dynatrace Research（Dynatrace研究院）；Institute of Advanced Research in Artificial Intelligence (IARAI)（高级人工智能研究所（IARAI））
In lifelong learning, an agent learns throughout its entire life without resets, in a constantly changing environment, as we humans do. Consequently, lifelong learning comes with a plethora of research problems such as continual domain shifts, which result in non-stationary rewards and environment dynamics. These non-stationarities are difficult to detect and cope with due to their continuous nature. Therefore, exploration strategies and learning methods are required that are capable of tracking the steady domain shifts, and adapting to them. We propose Reactive Exploration to track and react …
Beyond Forgetting: Diagnosing and Harnessing Shared Reasoning in Continual RLVR
State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室（BIGAI））；Beijing Institute of Technology（北京理工大学）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；State Key Lab of General AI, School of Intelligence Science and Technology, Peking University（通用人工智能国家重点实验室、北京大学智能科学与技术学院）
HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning
Technical University of Darmstadt（达姆施塔特工业大学）；Hessian Center for Artificial Intelligence (hessian.AI)（黑森人工智能中心）；Centre for Cognitive Science of Darmstadt（达姆施塔特认知科学中心）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
FutureWorld: A Live Reinforcement Learning Environment for Predictive Agents with Real-World Outcome Rewards
College of Software, Nankai University（南开大学软件学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；IIIS, Tsinghua University（清华大学智能系统与信息工程研究院）；Zhongguancun Academy, Beijing, China（北京中关村学院）
IDSIA USI-SUPSI, Università della Svizzera italiana（IDSIA USI-SUPSI，瑞士意大利语大学）；Technical University of Darmstadt（达姆施塔特工业大学）；Robotics Institute Germany (RIG)（德国机器人研究所（RIG））；German Research Center for AI (DFKI)（德国人工智能研究中心（DFKI））；University of Würzburg（维尔茨堡大学）；EPFL（洛桑联邦理工学院）；Politecnico di Milano（米兰理工大学）
Reward-free Pretraining for Reinforcement Learning via Occupancy Coverage Maximization
Computational Statistics and Machine Learning - Istituto Italiano di Tecnologia（计算统计与机器学习 - 意大利技术研究院）；AI Centre, Computer Science Department, University College London（人工智能中心，计算机科学系，伦敦大学学院）
The problem of balancing conflicting needs is fundamental to intelligence. Standard reinforcement learning algorithms maximize a scalar reward, which requires combining different objective-specific rewards into a single number. Alternatively, different objectives could also be combined at the level of action value, such that specialist modules responsible for different objectives submit different action suggestions to a decision process, each based on rewards that are independent of one another. In this work, we explore the potential benefits of this alternative strategy. We investigate a biol…
University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute（阿尔伯塔机器智能研究所）
In computational reinforcement learning, a growing body of work seeks to construct an agent's perception of the world through predictions of future sensations; predictions about environment observations are used as additional input features to enable better goal-directed decision-making. An open challenge in this line of work is determining from the infinitely many predictions that the agent could possibly make which predictions might best support decision-making. This challenge is especially apparent in continual learning problems where a single stream of experience is available to a singular…
Enhancing Reinforcement Learning for Radiology Report Generation with Evidence-aware Rewards and Self-correcting Preference Learning
Department of Computer Science and Engineering, ECUST（电子科技大学计算机科学与工程系）；Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, P. R. China（教育部能源化工过程智能制造重点实验室）；Zhejiang University（浙江大学）；Hangzhou High-Tech Zone (Binjiang) Institute of Blockchain and Data Security（杭州高新技术区（滨江）区块链与数据安全研究院）
RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（中国科学院计算技术研究所人工智能安全国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Xiaohongshu Inc.（小红书科技有限公司）
Department of Computer Science and Technology, Tsinghua University, China（计算机科学与技术系，清华大学，中国）；College of Computer Science, Zhejiang University, China（浙江大学计算机科学学院，中国）；College of Computer Science, Beijing University of Posts and Telecommunications, China（北京邮电大学计算机科学学院，中国）
本文提出连续GUI代理任务，通过引入GUI-Anchoring in Flux框架，解决GUI分布变化时持续学习稳定性问题，实验显示其优于现有方法。
Joint Continual Learning of Local Language Models and Cloud Offloading Decisions with Budget Constraints
Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN（电子与计算机工程学院，普渡大学）；Department of Computer Science, University of Exeter, UK（计算机科学系，埃克塞特大学）
Video2Reward: Generating Reward Function from Videos for Legged Robot Behavior Learning
Artificial Intelligence Research Institute, Shenzhen MSU-BIT University, China（人工智能研究院，深圳MSU-BIT大学，中国）；College of Mechatronics and Control Engineering, Shenzhen University（机电工程学院，深圳大学）；College of Computer Science and Software Engineering, Shenzhen University（计算机科学与软件工程学院，深圳大学）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（大数据系统计算技术国家工程实验室，深圳大学）；Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）
Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
The University of Hong Kong（香港大学）；The Hong Kong University of Science and Technology（香港科学与技术大学）；Hong Kong Generative AI Research and Development Center（香港生成式人工智能研究与开发中心）
CSRP: Chain-of-Thought Reasoning for Chinese Text Correction via Reinforcement Learning with Efficiency-Aware Rewards
School of Computer Science and Technology, East China Normal University（东华大学计算机科学与技术学院）；Shanghai Institute of Artificial Intelligence for Education, East China Normal University（东华大学教育人工智能研究所）
DeepMind（深度思维）；University of Alberta（阿尔伯塔大学）；Alberta Machine Intelligence Institute (Amii)（阿尔伯塔机器智能研究所）
Learned communication between agents is a powerful tool when approaching decision-making problems that are hard to overcome by any single agent in isolation. However, continual coordination and communication learning between machine agents or human-machine partnerships remains a challenging open problem. As a stepping stone toward solving the continual communication learning problem, in this paper we contribute a multi-faceted study into what we term Pavlovian signalling -- a process by which learned, temporally extended predictions made by one agent inform decision-making by another agent wit…
Towards Agentic Self-Learning LLMs in Search Environment
The Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences（认知与决策智能复杂系统实验室，自动化研究所，中国科学院）；School of Artificial Intelligence, University of Chinese Academy of Sciences（人工智能学院，中国科学院大学）；Xiaohongshu Inc（小红书公司）；Meituan Inc（美团公司）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）
