DistillLens: 通过Logit Lens实现对称的知识蒸馏
DistillLens: Symmetric Knowledge Distillation Through Logit Lens
AI总结：
DistillLens通过Logit Lens对称对齐学生和教师模型的思维过程，提升知识蒸馏性能。
摘要：
标准的知识蒸馏（KD）通过优化最终输出来压缩大型语言模型（LLMs），但通常将教师模型的中间层思维过程视为黑箱。虽然基于特征的知识蒸馏试图弥合这一差距，但现有方法（如MSE和不对称KL散度）忽略了最终输出所需的丰富不确定性轮廓。在本文中，我们引入了DistillLens，一种通过对称对齐学生和教师模型的演进思维过程的框架。通过将中间隐藏状态投影到词汇空间中，我们利用对称散度目标进行结构对齐。我们的分析证明，这种约束对双方施加双重惩罚，防止过自信和欠自信，同时保留对最终推断至关重要的高熵信息通道。在GPT-2和Llama架构上的广泛实验表明，DistillLens在多样化的指令遵循基准测试中始终优于标准KD和特征转移基线。代码可在https://github.com/manishdhakal/DistillLens上获得。
英文摘要：
Standard Knowledge Distillation (KD) compresses Large Language Models (LLMs) by optimizing final outputs, yet it typically treats the teacher's intermediate layer's thought process as a black box. While feature-based distillation attempts to bridge this gap, existing methods (e.g., MSE and asymmetric KL divergence) ignore the rich uncertainty profiles required for the final output. In this paper, we introduce DistillLens, a framework that symmetrically aligns the evolving thought processes of student and teacher models. By projecting intermediate hidden states into the vocabulary space via the Logit Lens, we enforce structural alignment using a symmetric divergence objective. Our analysis proves that this constraint imposes a dual-sided penalty, preventing both overconfidence and underconfidence while preserving the high-entropy information conduits essential for final deduction. Extensive experiments on GPT-2 and Llama architectures demonstrate that DistillLens consistently outperforms standard KD and feature-transfer baselines on diverse instruction-following benchmarks. The code is available at https://github.com/manishdhakal/DistillLens.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
