DWA-KD：双空间加权与时间扭曲对齐用于跨分词器知识蒸馏
DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation
AI总结：
DWA-KD通过双空间加权和时间扭曲对齐提升跨分词器知识蒸馏效果，实现更精确的词级和序列级对齐。
摘要：
知识蒸馏（KD）已成为压缩大型语言模型（LLMs）的关键技术。尽管现有的跨分词器KD方法已取得显著进展，但其效果仍受到序列和词汇层面次优对齐的限制。为了解决这些限制，我们引入了双空间加权和时间扭曲对齐（DWA-KD），一种新的跨分词器蒸馏框架，通过双空间熵基加权增强逐词蒸馏，并通过利用词汇和语义信息实现精确的序列级对齐。在词级别，DWA-KD将教师表示映射到学生空间并 vice versa，通过KL散度进行双空间KD。该过程由双空间权重调节，这些权重会加重学生不确定而教师自信的词，从而聚焦于信息丰富的词而不是同等对待所有位置。在序列级别，DWA-KD对嵌入和最终隐藏状态层应用Soft Dynamic Time Warping（Soft-DTW），使教师和学生序列之间的词汇和上下文语义实现稳健对齐。在各种NLP基准上的广泛实验表明，DWA-KD优于最先进的KD基线，而消融研究确认了基于熵的词加权和嵌入和最终隐藏状态层Soft-DTW对齐的互补贡献。
英文摘要：
Knowledge Distillation (KD) has emerged as a crucial technique for compressing Large Language Models (LLMs). Although existing cross-tokenizer KD methods have made notable progress, their effectiveness remains constrained by suboptimal alignment across sequence and vocabulary levels. To address these limitations, we introduce Dual-Space Weighting and Time-Warped Alignment (DWA-KD), a novel cross-tokenizer distillation framework that enhances token-wise distillation through dual-space entropy-based weighting and achieves precise sequence-level alignment by leveraging both lexical and semantic information. At the token level, DWA-KD maps teacher representations into the student space and vice versa, performing dual-space KD via Kullback-Leibler divergence (KL). The process is modulated by dual-space weights that up-weight tokens where the student is uncertain and the teacher is confident, thereby focusing learning on informative tokens rather than treating all positions equally. At the sequence level, DWA-KD applies Soft Dynamic Time Warping (Soft-DTW) to both the embedding and final hidden-state layers, enabling robust alignment of lexical and contextual semantics between teacher and student sequences. Extensive experiments across diverse NLP benchmarks demonstrate that DWA-KD outperforms state-of-the-art KD baselines, while ablation studies confirm the complementary contributions of entropy-based token weighting and embedding and final hidden state layer Soft-DTW alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
