AfroXLMR-Comet：面向低资源语言的多语言知识蒸馏与注意力匹配
AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages
AI总结：
本文提出AfroXLMR-Comet，一种结合知识蒸馏与注意力匹配的混合方法，用于压缩多语言模型，在五种非洲语言上以超85%的规模缩减保持教师模型85%以内的准确率，实现低资源环境下的高效部署。
摘要：
通过知识蒸馏进行语言模型压缩，已成为在资源受限环境中部署大型语言模型的一种有前景的方法。然而，现有方法在蒸馏多语言模型时，尤其是在低资源语言上，往往难以保持性能。在本文中，我们提出了一种新颖的混合蒸馏方法，将传统知识蒸馏与简化的注意力匹配机制相结合，专门针对多语言场景设计。我们的方法引入了一种极其紧凑的学生模型架构，其规模远小于传统的多语言模型。我们在五种非洲语言上评估了我们的方法：基尼亚卢旺达语、斯瓦希里语、豪萨语、伊博语和约鲁巴语。蒸馏得到的学生模型AfroXLMR-Comet成功捕获了更大教师模型（AfroXLMR-Large）的输出分布和内部注意力模式，同时将模型规模缩减了超过85%。实验结果表明，我们的混合方法在性能上与教师模型相比具有竞争力，保持了原始模型性能的85%以内的准确率，同时所需的计算资源大幅减少。我们的工作为在资源受限环境中部署高效多语言模型提供了一个实用框架，尤其有利于涉及非洲语言的应用。
英文摘要：
Language model compression through knowledge distillation has emerged as a promising approach for deploying large language models in resource-constrained environments. However, existing methods often struggle to maintain performance when distilling multilingual models, especially for low-resource languages. In this paper, we present a novel hybrid distillation approach that combines traditional knowledge distillation with a simplified attention matching mechanism, specifically designed for multilingual contexts. Our method introduces an extremely compact student model architecture, significantly smaller than conventional multilingual models. We evaluate our approach on five African languages: Kinyarwanda, Swahili, Hausa, Igbo, and Yoruba. The distilled student model; AfroXLMR-Comet successfully captures both the output distribution and internal attention patterns of a larger teacher model (AfroXLMR-Large) while reducing the model size by over 85%. Experimental results demonstrate that our hybrid approach achieves competitive performance compared to the teacher model, maintaining an accuracy within 85% of the original model's performance while requiring substantially fewer computational resources. Our work provides a practical framework for deploying efficient multilingual models in resource-constrained environments, particularly benefiting applications involving African languages.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
