认识自己，教导自己：面向选择性自蒸馏的内部信息流
Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation
- Peking University（北京大学）
- University of Oxford（牛津大学）
- The University of Hong Kong（香港大学）
机构由 AI 辅助整理，请以论文原文为准。
中文总结 AI 辅助
针对无外部教师的自蒸馏中信息传递未被度量的问题，提出检索引导的在线策略自蒸馏框架InFlow，通过度量信念偏移选择样本，在多个模型和知识域上取得最优平均性能。
中文摘要 AI 辅助
自蒸馏将知识蒸馏转化为一个封闭的学习循环，并为递归式自我改进提供了一条路径。然而，在没有外部教师的情况下，模型必须自行确定哪些信息能够改进其监督信号，以及哪些引发的变更应当被学习。现有方法通常孤立地改进教师生成的数据或选择训练样本，使得这些阶段之间传递的信息未被度量。我们提出了InFlow，一个检索引导的在线策略自蒸馏框架，将该过程建模为从潜在信息到实现信息的流动。InFlow首先利用基于确定性校准的隐藏状态轨迹检索潜在的信息源，然后通过教师初始答案信念与检索条件化答案信念之间的Jensen-Shannon散度来度量这些信息源的实际效果。具有较大信念偏移的样本被选择用于在线策略蒸馏。我们的分析形式化了检索与选择所优化的信息，并将答案层面的偏移与师生蒸馏差距联系起来。在四个开放权重语言模型和三个知识领域中，InFlow在比较的选择方法中取得了最强的跨模型平均性能，消融实验支持了该框架的两个阶段。我们的代码可在该https URL获取。
英文摘要
Self-distillation turns knowledge distillation into a closed learning loop and offers a path toward recursive self-improvement. Without an external teacher, however, the model must determine both what information can improve its supervision and which induced changes should be learned. Existing methods typically improve teacher-generated data or select training examples in isolation, leaving the information transferred between these stages unmeasured. We introduce InFlow, a retrieval-guided on-policy self-distillation framework that models this process as potential-to-realized information flow. InFlow first retrieves potentially informative sources using certainty-calibrated hidden-state trajectories, then measures their realized effect through the Jensen--Shannon divergence between the teacher's initial and retrieval-conditioned answer beliefs. Examples with larger belief shifts are selected for on-policy distillation. Our analysis formalizes the information optimized by retrieval and selection and relates the answer-level shift to the teacher--student distillation gap. Across four open-weight language models and three knowledge domains, InFlow achieves the strongest cross-model average among the compared selection methods, with ablations supporting both stages of the framework. Our code is available at https://github.com/1240148048/INFLOW.
