On-policy self-distillation (OPSD) improves the reasoning capabilities of large language models (LLMs) by providing dense token-level supervision for on-policy rollouts. However, existing OPSD methods often yield limited gains on complex reasoning tasks and suffer from severe training instability. We identify two key causes: conditioning the self-teacher on a complete verified solution encourages imitation of complete reference trajectories rather than extraction of transferable reasoning insights, while indiscriminate full-response distillation imposes superfluous supervision on already-valid reasoning prefixes. Together, these issues suppress reasoning diversity and contribute to late-stage mode collapse. We propose Reflective On-policy Self-Distillation (ROSD), which distills transferable reasoning insights rather than complete reference trajectories. For each erroneous rollout, a self-reflector contrasts it with a correct rollout from the same group to derive a corrective idea and identify the sentence containing the first reasoning error. The corrective idea provides the self-teacher with targeted guidance, while the diagnosed error boundary allows ROSD to mask out the distillation loss over the valid prefix and apply token-level distillation only from the first erroneous sentence onward. Experiments across multiple reasoning benchmarks and model backbones show that ROSD consistently outperforms standard OPSD and reinforcement learning baselines, better preserves reasoning diversity, stabilizes training, and mitigates late-stage mode collapse. Code is available at https://github.com/ZiqiZhao1/ROSD.
We consider the problem of imitation learning from a finite set of expert trajectories, without access to reinforcement signals. The classical approach of extracting the expert's reward function via inverse reinforcement learning, followed by reinforcement learning is indirect and may be computationally expensive. Recent generative adversarial methods based on matching the policy distribution between the expert and the agent could be unstable during training. We propose a new framework for imitation learning by estimating the support of the expert policy to compute a fixed reward function, whi…
Summarize-Exemplify-Reflect: Data-driven Insight Distillation Empowers LLMs for Few-shot Tabular Classification
ETH Zürich（苏黎世联邦理工学院）；University of Copenhagen（哥本哈根大学）；University of Science and Technology of China（中国科学技术大学）；University of Amsterdam（阿姆斯特丹大学）；East China Normal University（东华大学）
Beyond Imitation: Learning Key Reasoning Steps from Dual Chain-of-Thoughts in Reasoning Distillation
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）
The performance of autoregressive models on natural language generation tasks has dramatically improved due to the adoption of deep, self-attentive architectures. However, these gains have come at the cost of hindering inference speed, making state-of-the-art models cumbersome to deploy in real-world, time-sensitive settings. We develop a compression technique for autoregressive models that is driven by an imitation learning perspective on knowledge distillation. The algorithm is designed to address the exposure bias problem. On prototypical language generation tasks such as translation and su…
This paper proposes a new reinforcement learning (RL) algorithm that enhances exploration by amplifying the imitation effect (AIE). This algorithm consists of self-imitation learning and random network distillation algorithms. We argue that these two algorithms complement each other and that combining these two algorithms can amplify the imitation effect for exploration. In addition, by adding an intrinsic penalty reward to the state that the RL agent frequently visits and using replay memory for learning the feature state when using an exploration bonus, the proposed approach leads to deep ex…
Training automated agents to complete complex tasks in interactive environments is challenging: reinforcement learning requires careful hand-engineering of reward functions, imitation learning requires specialized infrastructure and access to a human expert, and learning from intermediate forms of supervision (like binary preferences) is time-consuming and extracts little information from each human intervention. Can we overcome these challenges by building agents that learn from rich, interactive feedback instead? We propose a new supervision paradigm for interactive learning based on "teacha…
Genetic algorithms have been widely used in many practical optimization problems. Inspired by natural selection, operators, including mutation, crossover and selection, provide effective heuristics for search and black-box optimization. However, they have not been shown useful for deep reinforcement learning, possibly due to the catastrophic consequence of parameter crossovers of neural networks. Here, we present Genetic Policy Optimization (GPO), a new genetic algorithm for sample-efficient deep policy optimization. GPO uses imitation learning for policy crossover in the state space and appli…
Structured Agent Distillation for Large Language Model
Carnegie Mellon University（卡内基梅隆大学）；Harvard University（哈佛大学）；MIT（麻省理工学院）；Northeastern University（东北大学）；Adobe Research（Adobe研究）；National University of Singapore（新加坡国立大学）；University of Georgia（佐治亚大学）；Florida International University（佛罗里达国际大学）
Hán Dān Xué Bù (Mimicry) or Qīng Chū Yú Lán (Mastery)? A Cognitive Perspective on Reasoning Distillation in Large Language Models
School of Philosophy, Anhui University（安徽大学哲学学院）；Institute of Neuroscience, Chinese Academy of Sciences（中国科学院神经科学研究所）；Faculty of Education, University of Cambridge（剑桥大学教育学院）；School of Foreign Studies, South China Normal University（华南师范大学外语学院）；Division of Psychology and Language Sciences, University College London（伦敦大学学院心理学与语言科学系）
University of Maryland（马里兰大学）；National Institute for Standards and Technology（美国国家标准与技术研究院）
We present Multiple Scenario Verifiable Reinforcement Learning via Policy Extraction (MSVIPER), a new method for policy distillation to decision trees for improved robot navigation. MSVIPER learns an "expert" policy using any Reinforcement Learning (RL) technique involving learning a state-action mapping and then uses imitation learning to learn a decision-tree policy from it. We demonstrate that MSVIPER results in efficient decision trees and can accurately mimic the behavior of the expert policy. Moreover, we present efficient policy distillation and tree-modification techniques that take ad…
Technische Universität Darmstadt（达姆施塔特工业大学）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
The control of nonlinear dynamical systems remains a major challenge for autonomous agents. Current trends in reinforcement learning (RL) focus on complex representations of dynamics and policies, which have yielded impressive results in solving a variety of hard control tasks. However, this new sophistication and extremely over-parameterized models have come with the cost of an overall reduction in our ability to interpret the resulting policies. In this paper, we take inspiration from the control community and apply the principles of hybrid switching systems in order to break down complex dy…
Skill-Conditioned Gated Self-Distillation for LLM Reasoning
Tsinghua University（清华大学）；Fudan University（复旦大学）；City University of Hong Kong（香港城市大学）；Huazhong University of Science and Technology（华中科技大学）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
RAPiD: Reward-Guided Consistency Distillation of Diffusion Planners for Real-Time Autonomous Driving
School of Information Technology, Monash University, Malaysia（墨尔本大学信息科技学院）；Faculty of Information Technology, Monash University, Australia（墨尔本大学信息技术学院）；TCS Research, India（印度TCS研究）
I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation
Allen Institute for AI（艾伦人工智能研究所）；University of Southern California（南加州大学）；Tohoku University（东北大学）；Northwestern University（西北大学）；University of Washington（华盛顿大学）
Information-Theoretic Distillation for Reference-less Summarization
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗 G. 艾伦计算机科学与工程学院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
Entropy-Aware On-Policy Distillation of Language Models
IBM Research, San Jose, CA, USA（IBM研究院，旧金山，加州，美国）；University of Toronto, Ontario, Canada（多伦多大学，安大略，加拿大）；Vector Institute, Ontario, Canada（向量研究所，安大略，加拿大）
Robust Reasoning via Dynamic Token Selection for Distribution-Aligned Self-Distillation
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University（北京未来区块链与隐私计算先进创新中心，北京航空航天大学）；School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）
Improving Retrospective Language Agents via Joint Policy Gradient Optimization
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）
What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situations
Paul G. Allen School of Computer Science & Engineering, University of Washington（华盛顿大学保罗·G·艾伦计算机科学与工程学院）；Allen Institute for Artificial Intelligence（艾伦人工智能研究所）
