通过教师语言选择与跨语言自蒸馏缓解多语言预训练语言模型（mPLMs）中的语言层面性能差异
Mitigating Language-Level Performance Disparity in mPLMs via Teacher Language Selection and Cross-lingual Self-Distillation
AI总结：
针对多语言预训练语言模型内部不同语言性能差异大的问题，提出无需额外标注数据的ALSACE方法，通过高性能语言知识指导低性能语言，有效缓解差异且在多类多语言NLU任务上表现优异。
摘要：
大规模多语言预训练语言模型（mPLMs）在跨语言任务上表现出色，但同一mPLM内部不同语言间存在显著的性能差异。过往研究试图通过用多语言数据对mPLMs进行有监督微调来缩小这些差异。然而，获取标注多语言数据耗时较长，且用有限的标注多语言数据微调mPLM只能封装标注数据特有的知识。因此，我们提出ALSACE方法，利用同一mPLM内部性能较好语言的已学习知识来指导性能较差的语言，无需额外的标注多语言数据。实验表明，ALSACE在多种mPLMs上有效缓解了语言层面的性能差异，同时在从全资源到有限资源设置的各类多语言自然语言理解（NLU）任务上表现出具有竞争力的性能。我们方法的代码可在https://github.com/pkunlp-icler/ALSACE 获取。
英文摘要：
Large-scale multilingual Pretrained Language Models (mPLMs) yield impressive performance on cross-language tasks, yet significant performance disparities exist across different languages within the same mPLM. Previous studies endeavored to narrow these disparities by supervise fine-tuning the mPLMs with multilingual data. However, obtaining labeled multilingual data is time-consuming, and fine-tuning mPLM with limited labeled multilingual data merely encapsulates the knowledge specific to the labeled data. Therefore, we introduce ALSACE to leverage the learned knowledge from the well-performing languages to guide under-performing ones within the same mPLM, eliminating the need for additional labeled multilingual data. Experiments show that ALSACE effectively mitigates language-level performance disparity across various mPLMs while showing the competitive performance on different multilingual NLU tasks, ranging from full resource to limited resource settings. The code for our approach is available at https://github.com/pkunlp-icler/ALSACE.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
