训练后概率流形校正通过结构化SVD剪枝与自参考知识蒸馏
Post-Training Probability Manifold Correction via Structured SVD Pruning and Self-Referential Distillation
AI总结：
SparseKD通过结构化SVD剪枝与自参考知识蒸馏，在训练后实现模型压缩，有效提升模型质量并减少参数量。
摘要：
大型语言模型部署成本高昂。我们引入了稀疏知识蒸馏（SparseKD），一种训练后方法，通过结合结构化SVD剪枝与自参考知识蒸馏来压缩Transformer模型。关键思想是简单：不使用外部教师，模型通过匹配压缩前的概率分布来教学自己。这种自参考设置在剧烈剪枝后实现了令人惊讶的质量恢复。我们的实验揭示了一个意外发现：在相同的客观函数和固定校准数据集下，仅使用自参考蒸馏，在训练后应用，相对于原始收敛检查点，模型质量提高了39%。当与结构化剪枝结合时，SparseKD在参数减少15-65%的同时，质量有可接受的权衡。核分析显示，速度提升完全来自于前馈层中减少的密集矩阵乘法，而注意力保持不变，使这种方法与注意力优化互补。我们在两个模型家族（0.6B和3.8B参数）上进行了验证，多种子实验确认了高可重复性。SparseKD不需要外部超教师，不需要架构更改，也不需要定制推理内核，使其能够立即使用现有基础设施部署。
英文摘要：
Large language models are expensive to deploy. We introduce Sparse Knowledge Distillation (SparseKD), a post-training method that compresses transformer models by combining structured SVD pruning with self-referential knowledge distillation. The key insight is simple: instead of using an external teacher, the model teaches itself by matching its own probability distribution from before compression. This self-referential setup enables surprisingly strong quality recovery after aggressive pruning. Our experiments reveal an unexpected finding: self-referential distillation alone, applied post-training under an identical objective and fixed calibration dataset, improves model quality by 39% relative to the original converged checkpoint. When combined with structured pruning, SparseKD achieves 15-65% parameter reduction with acceptable quality trade-offs. Kernel profiling shows that speedups arise entirely from reduced dense matrix multiplication in feed-forward layers while attention remains unchanged, making this approach complementary to attention optimizations. We validate across two model families (0.6B and 3.8B parameters) with multi-seed experiments confirming high reproducibility. SparseKD requires no external super-teacher, no architectural changes, and no custom inference kernels, making it immediately deployable with existing infrastructure.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
