arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.12413cs.CL
收藏

面向低资源语系的定向多语言适配

Targeted Multilingual Adaptation for Low-resource Language Families

University of Washington(华盛顿大学);Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学保罗·G·艾伦计算机科学与工程学院)

C. M. Downey, Terra Blevins, Dhwani Serai, Dwija Parikh, Shane Steinert-Threlkeld

AI总结:

该研究以乌拉尔语系为案例,系统探索预训练模型定向适配低资源语系的最佳实践,验证了亲缘语言训练的增益,得出词表大小影响弱、低资源语言可大幅上采样等结论,提升了模型在低资源语言上的性能。

摘要:

众所周知,多语言模型的“大规模多语言”训练会限制其在任一单语言上的效用,且在低资源语言上表现尤为糟糕。但有证据表明,低资源语言可从定向多语言处理中获益——即模型在亲缘关系相近的语言上训练。为更严谨地验证该方法,我们系统研究了将预训练模型适配到某一语系的最佳实践。我们以乌拉尔语系为测试案例,在多种配置下适配XLM-R以建模15种语言;随后在两项下游任务和11种评估语言上评测各实验设置的性能。我们的适配模型显著优于单语言和多语言基线。此外,对超参数效应的回归分析显示,适配后的词表大小对低资源语言而言相对不重要,且训练期间可对低资源语言进行大幅上采样,而对高资源语言的性能几乎没有损害。这些结果为定向场景下的语言适配提出了新的最佳实践。

英文摘要:

The "massively-multilingual" training of multilingual models is known to limit their utility in any one language, and they perform particularly poorly on low-resource languages. However, there is evidence that low-resource languages can benefit from targeted multilinguality, where the model is trained on closely related languages. To test this approach more rigorously, we systematically study best practices for adapting a pre-trained model to a language family. Focusing on the Uralic family as a test case, we adapt XLM-R under various configurations to model 15 languages; we then evaluate the performance of each experimental setting on two downstream tasks and 11 evaluation languages. Our adapted models significantly outperform mono- and multilingual baselines. Furthermore, a regression analysis of hyperparameter effects reveals that adapted vocabulary size is relatively unimportant for low-resource languages, and that low-resource languages can be aggressively up-sampled during training at little detriment to performance in high-resource languages. These results introduce new best practices for performing language adaptation in a targeted setting.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑