ACTD：带残差正则化的基于锚点的跨分词器蒸馏
ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization
浏览论文内容
中文总结 AI 辅助
针对跨分词器蒸馏的词汇与序列不对齐及对齐噪声问题，提出带残差正则化的ACTD方法，在五个推理基准上用三种教师模型评估取得SOTA，多教师扩展版性能优于基线。
中文摘要 AI 辅助
知识蒸馏可有效将推理能力从大型语言模型迁移至轻量学生模型。为实现跨不同模型家族的知识迁移，研究者日益探索跨分词器蒸馏。然而，跨分词器蒸馏因词汇与序列不对齐仍具挑战性，而近似词汇对齐会向蒸馏引入额外噪声。为应对这些挑战，我们提出带残差正则化的基于锚点的跨分词器蒸馏（ACTD）。ACTD通过词汇与序列对齐弥合结构异质性，同时通过带残差正则化的新型锚点损失缓解对齐噪声。我们进一步将该框架扩展至多教师设置。在五个推理基准上采用三种不同教师模型评估，ACTD取得了最先进的性能。此外，其多教师扩展版本优于最强的单教师与多教师基线，进一步证明了我们方法的鲁棒性。
英文摘要
Knowledge distillation effectively transfers reasoning capabilities from large language models to lightweight student models. To enable knowledge transfer across disparate model families, researchers increasingly explore cross-tokenizer distillation. However, cross-tokenizer distillation remains challenging due to vocabulary and sequence misalignment, while approximate vocabulary alignment can introduce additional noise into distillation. To address these challenges, we propose Anchor-Based Cross-Tokenizer Distillation with Residual Regularization (ACTD). ACTD bridges structural heterogeneity through vocabulary and sequence alignment, while mitigating alignment noise via a novel anchor loss with residual regularization. We further extend this framework to a multi-teacher setting. Evaluated across five reasoning benchmarks with three distinct teacher models, ACTD achieves state-of-the-art performance. Moreover, its multi-teacher extension outperforms the strongest single-teacher and multi-teacher baselines, further demonstrating the robustness of our method.
发表机构
- Harbin Institute of Technology（哈尔滨工业大学）
- Peng Cheng Laboratory（鹏城实验室）
机构由 AI 辅助整理，请以论文原文为准。
