We propose RLAnything, a reinforcement learning framework that dynamically forges environment, policy, and reward models through closed-loop optimization, amplifying learning signals and strengthening the overall RL system for any LLM or agentic scenarios. Specifically, the policy is trained with integrated feedback from step-wise and outcome signals, while the reward model is jointly optimized via consistency feedback, which in turn further improves policy training. Moreover, our theory-motivated automatic environment adaptation improves training for both the reward and policy models by leveraging critic feedback from each, enabling learning from experience. Empirically, each added component consistently improves the overall system, and RLAnything yields substantial gains across various representative LLM and agentic tasks, boosting Qwen3-VL-8B-Thinking by 9.1% on OSWorld and Qwen2.5-7B-Instruct by 18.7% and 11.9% on AlfWorld and LiveBench, respectively. We also that optimized reward-model signals outperform outcomes that rely on human labels. Code: https://github.com/Gen-Verse/Open-AgentRL
OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL
School of Computer Science and Information Engineering, Hefei University of Technology, Hefei, China（合肥工业大学计算机科学与信息工程学院）；Wuhan University, Wuhan, China（武汉大学）；Lab for Intelligence and visiON (LION)（智能视觉实验室）
Forge: Quality-Aware Reinforcement Learning for NP-Hard Optimization in LLMs
Tongji University（同济大学）；Shanghai AI Lab（上海人工智能实验室）；Zhejiang University（浙江大学）；Fudan University（复旦大学）；The Chinese University of Hong Kong（香港中文大学）；Independent（独立）
EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
The Chinese University of Hong Kong（香港中文大学）；Microsoft Research Asia（微软亚洲研究院）；Hong Kong Baptist University（香港浸会大学）；LMU Munich（慕尼黑大学）；MoE Key Laboratory of High Confidence Software Technologies（教育部高可信软件技术重点实验室）
ETH Zürich（苏黎世联邦理工学院）；John A. Paulson School of Engineering and Applied Sciences（约翰·A·保尔森工程与应用科学学院）；Harvard University（哈佛大学）
We present the extension of the Remember and Forget for Experience Replay (ReF-ER) algorithm to Multi-Agent Reinforcement Learning (MARL). ReF-ER was shown to outperform state of the art algorithms for continuous control in problems ranging from the OpenAI Gym to complex fluid flows. In MARL, the dependencies between the agents are included in the state-value estimator and the environment dynamics are modeled via the importance weights used by ReF-ER. In collaborative environments, we find the best performance when the value is estimated using individual rewards and we ignore the effects of ot…
Exploration bonus derived from the novelty of the states in an environment has become a popular approach to motivate exploration for deep reinforcement learning agents in the past few years. Recent methods such as curiosity-driven exploration usually estimate the novelty of new observations by the prediction errors of their system dynamics models. Due to the capacity limitation of the models and difficulty of performing next-frame prediction, however, these methods typically fail to balance between exploration and exploitation in high-dimensional observation tasks, resulting in the agents forg…
Lifelong learning is a very important step toward realizing robust autonomous artificial agents. Neural networks are the main engine of deep learning, which is the current state-of-the-art technique in formulating adaptive artificial intelligent systems. However, neural networks suffer from catastrophic forgetting when stressed with the challenge of continual learning. We investigate how to exploit modular topology in neural networks in order to dynamically balance the information load between different modules by routing inputs based on the information content in each module so that informati…
Reward-free, unsupervised discovery of skills is an attractive alternative to the bottleneck of hand-designing rewards in environments where task supervision is scarce or expensive. However, current skill pre-training methods, like many RL techniques, make a fundamental assumption - stationary environments during training. Traditional methods learn all their skills simultaneously, which makes it difficult for them to both quickly adapt to changes in the environment, and to not forget earlier skills after such adaptation. On the other hand, in an evolving or expanding environment, skill learnin…
Although reinforcement learning (RL) can provide reliable solutions in many settings, practitioners are often wary of the discrepancies between the RL solution and their status quo procedures. Therefore, they may be reluctant to adapt to the novel way of executing tasks proposed by RL. On the other hand, many real-world problems require relatively small adjustments from the status quo policies to achieve improved performance. Therefore, we propose a student-teacher RL mechanism in which the RL (the "student") learns to maximize its reward, subject to a constraint that bounds the difference bet…
Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling
Huawei Noah’s Ark Lab, Montréal, Canada（华为诺亚实验室，加拿大蒙特利尔）；McGill University and Mila - Québec AI Institute（麦吉尔大学和魁北克人工智能研究所）；Huawei Noah’s Ark Lab, Beijing, China（华为诺亚实验室，中国北京）
Replay Buffer with Local Forgetting for Adapting to Local Environment Changes in Deep Model-Based Reinforcement Learning
Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）；Université de Montréal（蒙特利尔大学）；Microsoft Research（微软研究院）；École Polytechnique de Montréal（蒙特利尔高等理工学院）；Canada CIFAR AI Chair（加拿大CIFAR人工智能讲席）
Reinforcement Learning with Semantic Rewards Enables Low-Resource Language Expansion without Alignment Tax
Minzu University of China（中国民族大学）；Ant Group（蚂蚁集团）；Shanghai Jiao Tong University（上海交通大学）；University of Macau（澳门大学）；Peking University（北京大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Hainan International College, Minzu University of China（中国民族大学海南国际学院）
We learn end-to-end point-to-point and path-following navigation behaviors that avoid moving obstacles. These policies receive noisy lidar observations and output robot linear and angular velocities. The policies are trained in small, static environments with AutoRL, an evolutionary automation layer around Reinforcement Learning (RL) that searches for a deep RL reward and neural network architecture with large-scale hyper-parameter optimization. AutoRL first finds a reward that maximizes task completion, and then finds a neural network architecture that maximizes the cumulative of the found re…
Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；University of Glasgow（格拉斯哥大学）
From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation
Huawei Noah’s Ark Lab（华为诺亚实验室）；Department of Data Science, EURECOM（EURECOM数据科学系）；Cognizant AI Lab（Cognizant人工智能实验室）；Statistics Program, KAUST（KAUST统计学项目）
Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；JD.COM（京东）
Continual Learning (CL) considers the problem of training an agent sequentially on a set of tasks while seeking to retain performance on all previous tasks. A key challenge in CL is catastrophic forgetting, which arises when performance on a previously mastered task is reduced when learning a new task. While a variety of methods exist to combat forgetting, in some cases tasks are fundamentally incompatible with each other and thus cannot be learnt by a single policy. This can occur, in reinforcement learning (RL) when an agent may be rewarded for achieving different goals from the same observa…
Fusing Bidirectional Chains of Thought and Reward Mechanisms A Method for Enhancing Question-Answering Capabilities of Large Language Models for Chinese Intangible Cultural Heritage
Carnegie Mellon University（卡内基梅隆大学）；Amazon AI（亚马逊人工智能）；University of California San Diego（加利福尼亚大学圣地亚哥分校）；Google（谷歌）；Microsoft Research（微软研究院）；Citadel（西得尔）
Many practical environments contain catastrophic states that an optimal agent would visit infrequently or never. Even on toy problems, Deep Reinforcement Learning (DRL) agents tend to periodically revisit these states upon forgetting their existence under a new policy. We introduce intrinsic fear (IF), a learned reward shaping that guards DRL agents against periodic catastrophes. IF agents possess a fear model trained to predict the probability of imminent catastrophe. This score is then used to penalize the Q-learning objective. Our theoretical analysis bounds the reduction in average return …
Weights-Rotated Preference Optimization for Large Language Models
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）；Baidu Inc.（百度公司）
The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
Fudan University（复旦大学）；INFLY TECH（INFLY科技）；Griffith University（格里菲斯大学）；University of Waterloo（滑铁卢大学）；Shanghai Innovation Institute（上海创新研究院）；Shanghai Academy of Artificial Intelligence for Science（上海人工智能科学研究院）
Real-world applications of reinforcement learning for recommendation and experimentation faces a practical challenge: the relative reward of different bandit arms can evolve over the lifetime of the learning agent. To deal with these non-stationary cases, the agent must forget some historical knowledge, as it may no longer be relevant to minimise regret. We present a solution to handling non-stationarity that is suitable for deployment at scale, to provide business operators with automated adaptive optimisation. Our solution aims to provide interpretable learning that can be trusted by humans,…
The promise of reinforcement learning is to solve complex sequential decision problems autonomously by specifying a high-level reward function only. However, reinforcement learning algorithms struggle when, as is often the case, simple and intuitive rewards provide sparse and deceptive feedback. Avoiding these pitfalls requires thoroughly exploring the environment, but creating algorithms that can do so remains one of the central challenges of the field. We hypothesise that the main impediment to effective exploration originates from algorithms forgetting how to reach previously visited states…
UC Berkeley（加州大学伯克利分校）；University of Amsterdam（阿姆斯特丹大学）；Stanford University（斯坦福大学）
Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not only specify what to do, but also the much larger space of what not to do. It is easy to forget these preferences, since these preferences are already satisfied in our environment. This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want. We can therefore use this implicit preference information from the state to…
Tutor-Student Reinforcement Learning: A Dynamic Curriculum for Robust Deepfake Detection
School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Integrated Circuits, Peking University（北京大学集成电路学院）；School of Information, Huazhong Agricultural University（华中农业大学信息学院）；Cyberspace Institute of Advanced Technology, Guangzhou University（广州大学先进技术网络研究院）
