数据与平衡可以兼得：迈向平衡且高效的多语言模型
You Can Have Your Data and Balance It Too: Towards Balanced and Efficient Multilingual Models
AI总结：
本文提出基于师生知识蒸馏的多语言训练方法，利用单语教师模型和平衡子采样数据训练单一多语言学生模型，在保持高资源语言性能的同时提升低资源语言表现。
摘要：
多语言模型已被广泛用于向低资源语言进行跨语言迁移。然而，这些语言在预训练数据中的代表性不足，阻碍了其性能表现。为缓解这一问题，我们提出了一种基于师生知识蒸馏的新型多语言训练技术。在该设置中，我们利用针对各自语言优化的单语教师模型。我们使用这些教师模型以及平衡（子采样）的数据，将教师的知识蒸馏到一个单一的多语言学生模型中。我们的方法在低资源语言上优于标准训练方法，并在使用相同数据量的情况下保持了高资源语言上的性能。若得到广泛应用，我们的方法可以提高低资源语言在自然语言处理系统中的代表性。
英文摘要：
Multilingual models have been widely used for cross-lingual transfer to low-resource languages. However, the performance on these languages is hindered by their underrepresentation in the pretraining data. To alleviate this problem, we propose a novel multilingual training technique based on teacher-student knowledge distillation. In this setting, we utilize monolingual teacher models optimized for their language. We use those teachers along with balanced (sub-sampled) data to distill the teachers' knowledge into a single multilingual student. Our method outperforms standard training methods in low-resource languages and retrains performance on high-resource languages while using the same amount of data. If applied widely, our approach can increase the representation of low-resource languages in NLP systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
