面向低资源语言的多语言编码器语言模型压缩研究
On Multilingual Encoder Language Model Compression for Low-Resource Languages
AI总结：
本文提出一种结合知识蒸馏、结构化剪枝、截断和词表裁剪的极致压缩方法，将多语言编码器模型压缩为低资源语言的单语言小模型，在保持竞争力的同时实现高达92%的压缩率。
摘要：
在本文中，我们结合两步知识蒸馏、结构化剪枝、截断和词表裁剪，对面向低资源语言的多语言仅编码器语言模型进行极致压缩。我们的新方法系统性地组合现有技术并将其推向极限，通过减少层深度、前馈隐藏层大小和中间层嵌入尺寸，创建显著更小的单语言模型，同时保留关键的语言特定知识。我们实现了高达92%的压缩率，同时保持有竞争力的性能，在三个低资源语言的四个下游任务（包括情感分析、主题分类、命名实体识别和词性标注）中，中等压缩下平均性能下降2-10%，最大压缩下下降8-13%。值得注意的是，性能退化与教师模型中语言特定数据的数量相关，数据量越大，性能损失越小。此外，我们进行了消融研究，以确定使用这些技术进行多语言模型压缩的最佳实践。
英文摘要：
In this paper, we combine two-step knowledge distillation, structured pruning, truncation, and vocabulary trimming for extremely compressing multilingual encoder-only language models for low-resource languages. Our novel approach systematically combines existing techniques and takes them to the extreme, reducing layer depth, feed-forward hidden size, and intermediate layer embedding size to create significantly smaller monolingual models while retaining essential language-specific knowledge. We achieve compression rates of up to 92% while maintaining competitive performance, with average drops of 2-10% for moderate compression and 8-13% at maximum compression in four downstream tasks, including sentiment analysis, topic classification, named entity recognition, and part-of-speech tagging, across three low-resource languages. Notably, the performance degradation correlates with the amount of language-specific data in the teacher model, with larger datasets resulting in smaller performance losses. Additionally, we conduct ablation studies to identify the best practices for multilingual model compression using these techniques.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
