Reinforcement learning (RL) post-training has recently driven major gains in long chain-of-thought reasoning large language models (LLMs), but the high inference cost of such models motivates distillation into smaller students. Most existing knowledge distillation (KD) methods are designed for supervised fine-tuning (SFT), relying on fixed teacher traces or teacher-student Kullback-Leibler (KL) divergence-based regularization. When combined with RL, these approaches often suffer from distribution mismatch and objective interference: teacher supervision may not align with the student's evolving rollout distribution, and the KL regularizer can compete with reward maximization and require careful loss balancing. To address these issues, we propose RL-aware distillation (RLAD), which performs selective imitation during RL -- guiding the student toward the teacher only when it improves the current policy update. Our core component, Trust Region Ratio Distillation (TRRD), replaces the teacher-student KL regularizer with a PPO/GRPO-style likelihood-ratio objective anchored to a teacher--old-policy mixture, yielding advantage-aware, trust-region-bounded distillation on student rollouts and naturally balancing exploration, exploitation, and imitation. Across diverse logic reasoning and math benchmarks, RLAD consistently outperforms offline distillation, standard GRPO, and KL-based on-policy teacher-student knowledge distillation.
Teacher-Guided Policy Optimization for On-Policy Reasoning Distillation under Large Policy Divergence
School of Computer Science and Engineering, Northeastern University, China（东北大学计算机科学与工程学院）；Tsinghua University（清华大学）；Meituan（美团）；Meta AI；NiuTrans Research, Shenyang, China（新译研究院，沈阳，中国）
Reasoning Compression with Mixed-Policy Distillation
Technical University of Munich（慕尼黑技术大学）；GESIS – Leibniz Institute for the Social Sciences（莱布尼茨社会科学研究所）；Northeastern University（东北大学）；LMU Munich（慕尼黑大学）；Siemens AG（西门子股份公司）；University of Cambridge（剑桥大学）；University of Oxford（牛津大学）；Mina AI
Federated Reasoning Distillation Framework with Model Learnability-Aware Data Allocation
Beihang University（北航）；Heilongjiang University（黑龙江大学）；Beijing University of Civil Engineering and Architecture（北京建筑大学）；Yanshan University（燕山大学）；Shandong University（山东大学）；Renmin University of China（中国人民大学）
Learning to Revise Reasoning with Segment-wise On-Policy Distillation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；University of Science and Technology of China（中国科学技术大学）；Alibaba Group（阿里巴巴集团）；Stanford University（斯坦福大学）
Hán Dān Xué Bù (Mimicry) or Qīng Chū Yú Lán (Mastery)? A Cognitive Perspective on Reasoning Distillation in Large Language Models
School of Philosophy, Anhui University（安徽大学哲学学院）；Institute of Neuroscience, Chinese Academy of Sciences（中国科学院神经科学研究所）；Faculty of Education, University of Cambridge（剑桥大学教育学院）；School of Foreign Studies, South China Normal University（华南师范大学外语学院）；Division of Psychology and Language Sciences, University College London（伦敦大学学院心理学与语言科学系）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Amazon.com Inc（亚马逊公司）；Georgia Institute of Technology（佐治亚理工学院）
This paper investigates cross-lingual temporal knowledge graph reasoning problem, which aims to facilitate reasoning on Temporal Knowledge Graphs (TKGs) in low-resource languages by transfering knowledge from TKGs in high-resource ones. The cross-lingual distillation ability across TKGs becomes increasingly crucial, in light of the unsatisfying performance of existing reasoning methods on those severely incomplete TKGs, especially in low-resource languages. However, it poses tremendous challenges in two aspects. First, the cross-lingual alignments, which serve as bridges for knowledge transfer…
Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation
College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院）；Key Laboratory of Symbolic Computation and Knowledge Engineering, MoE, Jilin University（吉林大学符号计算与知识工程重点实验室）；Tongyi Lab, Alibaba Group（阿里集团通义实验室）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）；School of Artificial Intelligence, Jilin University（吉林大学人工智能学院）
Key-Point-Driven Mathematical Reasoning Distillation of Large Language Model
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；School of Artificial Intelligence, Beijing Normal University（北京师范大学人工智能学院）
National Tsing Hua University（国立清华大学）；Preferred Networks（优选网络）
Off-policy ensemble reinforcement learning (RL) methods have demonstrated impressive results across a range of RL benchmark tasks. Recent works suggest that directly imitating experts' policies in a supervised manner before or during the course of training enables faster policy improvement for an RL agent. Motivated by these recent insights, we propose Periodic Intra-Ensemble Knowledge Distillation (PIEKD). PIEKD is a learning framework that uses an ensemble of policies to act in the environment while periodically sharing knowledge amongst policies in the ensemble through knowledge distillatio…
We propose Algorithm Distillation (AD), a method for distilling reinforcement learning (RL) algorithms into neural networks by modeling their training histories with a causal sequence model. Algorithm Distillation treats learning to reinforcement learn as an across-episode sequential prediction problem. A dataset of learning histories is generated by a source RL algorithm, and then a causal transformer is trained by autoregressively predicting actions given their preceding learning histories as context. Unlike sequential policy prediction architectures that distill post-learning or expert sequ…
Yonsei University（延世大学）；Deakin University（迪肯大学）；Samsung Electronics Co., Ltd.（三星电子株式会社）；University of Oulu（奥卢大学）
Traditional distributed deep reinforcement learning (RL) commonly relies on exchanging the experience replay memory (RM) of each agent. Since the RM contains all state observations and action policy history, it may incur huge communication overhead while violating the privacy of each agent. Alternatively, this article presents a communication-efficient and privacy-preserving distributed RL framework, coined federated reinforcement distillation (FRD). In FRD, each agent exchanges its proxy experience replay memory (ProxRM), in which policies are locally averaged with respect to proxy states clu…
University of South Carolina（南卡罗来纳大学）；San Francisco State University（旧金山州立大学）
Policy distillation in deep reinforcement learning provides an effective way to transfer control policies from a larger network to a smaller untrained network without a significant degradation in performance. However, policy distillation is underexplored in deep reinforcement learning, and existing approaches are computationally inefficient, resulting in a long distillation time. In addition, the effectiveness of the distillation process is still limited to the model capacity. We propose a new distillation mechanism, called real-time policy distillation, in which training the teacher model and…
Tsinghua University（清华大学）；Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
The powerful learning ability of deep neural networks enables reinforcement learning agents to learn competent control policies directly from continuous environments. In theory, to achieve stable performance, neural networks assume i.i.d. inputs, which unfortunately does no hold in the general reinforcement learning paradigm where the training data is temporally correlated and non-stationary. This issue may lead to the phenomenon of "catastrophic interference" and the collapse in performance. In this paper, we present IQ, i.e., interference-aware deep Q-learning, to mitigate catastrophic inter…
