Reinforcement Learning is commonly used to train large language models using environmental feedback. In applied settings, the environment usually provides sparse or delayed feedback. This makes it difficult for the model to pinpoint which actions in its reasoning led to success or failure. So, learning effectively from these signals is hard because the model must determine how each failure should inform meaningful behavioral corrections in subsequent iterations. We introduce a training framework, Self-Review Reinforcement Learning, that embeds an explicit self-review step into each RL episode. When a first-pass response fails, the model generates a self-review to identify what went wrong, which conditions an improved second attempt. Unlike inference-time reflection approaches, such as Reflexion, the framework optimizes self-review with policy gradients and internalizes improvements into the base policy via selective distillation, ensuring they persist across future episodes. A cross-episode memory keeps successful self-reviews for reuse when encountering similar tasks in future episodes during training. We evaluate SRRL against a standard RLVR baseline using the GRPO optimizer across two language models, Qwen 3-4B and OLMo-3- 7B, on GSM8K benchmark. SRRL consistently outperforms the RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement.
We propose Algorithm Distillation (AD), a method for distilling reinforcement learning (RL) algorithms into neural networks by modeling their training histories with a causal sequence model. Algorithm Distillation treats learning to reinforcement learn as an across-episode sequential prediction problem. A dataset of learning histories is generated by a source RL algorithm, and then a causal transformer is trained by autoregressively predicting actions given their preceding learning histories as context. Unlike sequential policy prediction architectures that distill post-learning or expert sequ…
Many applications of reinforcement learning can be formalized as goal-conditioned environments, where, in each episode, there is a "goal" that affects the rewards obtained during that episode but does not affect the dynamics. Various techniques have been proposed to improve performance in goal-conditioned environments, such as automatic curriculum generation and goal relabeling. In this work, we explore a connection between off-policy reinforcement learning in goal-conditioned settings and knowledge distillation. In particular: the current Q-value function and the target Q-value estimate are b…
PC3D: Zero-Shot Cooperation Across Variable Rosters via Personalized Context Distillation
Doctoral School of Exact and Natural Sciences, Jagiellonian University（杰哥利昂大学精确与自然科学博士学院）；Faculty of Mathematics and Computer Science, Jagiellonian University（杰哥利昂大学数学与计算机科学学院）
PR-OPD: Privileged Representation On-policy Self-Distillation for Agentic Reinforcement Learning
University of Florida（佛罗里达大学）；University of Illinois at Chicago（伊利诺伊大学芝加哥分校）；Case Western Reserve University（凯斯西储大学）；Vanderbilt University（范德堡大学）；Weill Cornell Medicine（威尔康奈尔医学院）
Huawei UK R&D（华为英国研发中心）；University of Cambridge（剑桥大学）；University College London（伦敦大学学院）
We consider a context-dependent Reinforcement Learning (RL) setting, which is characterized by: a) an unknown finite number of not directly observable contexts; b) abrupt (discontinuous) context changes occurring during an episode; and c) Markovian context evolution. We argue that this challenging case is often met in applications and we tackle it using a Bayesian approach and variational inference. We adapt a sticky Hierarchical Dirichlet Process (HDP) prior for model learning, which is arguably best-suited for Markov process modeling. We then derive a context distillation procedure, which id…
Georgia Institute of Technology（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）
This study proposes a safe and sample-efficient reinforcement learning (RL) framework to address two major challenges in developing applicable RL algorithms: satisfying safety constraints and efficiently learning with limited samples. To guarantee safety in real-world complex environments, we use the safe set algorithm (SSA) to monitor and modify the nominal controls, and evaluate SSA+RL in a clustered dynamic environment which is challenging to be solved by existing RL algorithms. However, the SSA+RL framework is usually not sample-efficient especially in reward-sparse environments, which has…
University of California, Los Angeles（加州大学洛杉矶分校）；Microsoft Research（微软研究院）
We study lifelong reinforcement learning (RL) in a regret minimization setting of linear contextual Markov decision process (MDP), where the agent needs to learn a multi-task policy while solving a streaming sequence of tasks. We propose an algorithm, called UCB Lifelong Value Distillation (UCBlvd), that provably achieves sublinear regret for any sequence of tasks, which may be adaptively chosen based on the agent's past behaviors. Remarkably, our algorithm uses only sublinear number of planning calls, which means that the agent eventually learns a policy that is near optimal for multiple task…
UniOPSD: Unifying Outcome and Hindsight Feedback for Agentic Reinforcement Learning
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Peking University（北京大学）；Mininglamp Technology（明略科技）；Key Laboratory of Computing Power Network and Information Security, Ministry of Education（计算力网络与信息安全教育部重点实验室；山东省计算中心（齐鲁工业大学（山东省科学院）））；Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences)（计算力互联网与服务计算重点实验室，山东省计算机科学基础研究中心）；Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science
DistilCLIP-EEG: Enhancing Epileptic Seizure Detection Through Multi-modal Learning and Knowledge Distillation
Aliyun School of Big Data, Changzhou University（阿里云大数据学院，长洲大学）；Department of Computing, Xi’an JiaoTong-Liverpool University（计算系，西安交通大学-利物浦大学）；Department of Computer Science, University of Liverpool（计算机科学系，利物浦大学）；Center for Artificial Intelligence in Agriculture, Fujian Agriculture and Forestry University（农业人工智能中心，福建农林大学）
Graph-Conditioned On-Policy Agent Distillation from Off-the-Shelf Teachers
Yuanbao Team, Tencent（腾讯元宝团队）；Tsinghua University（清华大学）；Huazhong University of Science and Technology（华中科技大学）；School of Artificial Intelligence, Shanghai Jiao Tong University（上海交通大学人工智能学院）
Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamwork
C$^{2}$DL, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所C²DL实验室）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；School of Future Technology, University of Chinese Academy of Sciences（中国科学院大学未来技术学院）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；University of Science and Technology of China（中国科学技术大学）；Qwen Team, Alibaba Group（阿里集团Qwen团队）
