特权学生：论初始化在多语言知识蒸馏中的价值
The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation
AI总结：
本文研究多语言知识蒸馏中初始化的重要性，提出直接复制教师模型权重以增强初始化，实验表明该方法比蒸馏过程本身贡献更大，且能保留低资源场景下的多语言能力。
摘要：
知识蒸馏（KD）已被证明是在许多自然语言处理任务中提升较小模型性能的成功策略。然而，大多数知识蒸馏工作仅探索单语场景。在本文中，我们研究了知识蒸馏在多语言环境中的价值。通过分析学生模型从教师模型中获取多语言知识的效果，我们发现知识蒸馏和模型初始化的重要性。我们提出的方法强调直接将教师模型的权重复制到学生模型以增强初始化。我们的研究结果表明，在各种多语言设置下，使用来自微调后教师模型的复制权重进行模型初始化，相比蒸馏过程本身贡献最大。此外，我们证明即使在低资源场景下，高效的权重初始化也能保留多语言能力。
英文摘要：
Knowledge distillation (KD) has proven to be a successful strategy to improve the performance of smaller models in many NLP tasks. However, most of the work in KD only explores monolingual scenarios. In this paper, we investigate the value of KD in multilingual settings. We find the significance of KD and model initialization by analyzing how well the student model acquires multilingual knowledge from the teacher model. Our proposed method emphasizes copying the teacher model's weights directly to the student model to enhance initialization. Our findings show that model initialization using copy-weight from the fine-tuned teacher contributes the most compared to the distillation process itself across various multilingual settings. Furthermore, we demonstrate that efficient weight initialization preserves multilingual capabilities even in low-resource scenarios.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
