arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.02534cs.CLcs.AI
收藏

ANGOFA: 利用OFA嵌入初始化和合成数据进行安哥拉语言模型

ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model

Masakhane NLP;Department of Computer Science University College London(计算机科学系伦敦大学学院);University College London(伦敦大学学院)

Osvaldo Luamba Quinjica, David Ifeoluwa Adelani

AI总结:

本文提出四个针对安哥拉语言定制的预训练语言模型,采用多语言自适应微调方法,通过有意识的嵌入初始化和合成数据提升模型性能,优于SOTA AfroXLMR-base和OFA模型。

摘要:

近年来,预训练语言模型(PLMs)的发展势头迅猛,展现出跨越语言障碍和促进知识转移的能力。然而,这一进展主要忽略了非常低资源语言,导致多语言领域存在明显空白。本文通过引入四个专门针对安哥拉语言微调的PLM,采用多语言自适应微调(MAFT)方法来填补这一空白。本文探讨了有意识的嵌入初始化和合成数据在提升MAFT模型下游任务性能中的作用。我们通过12.3和3.8个点分别提升了基线SOTA AfroXLMR-base(通过MAFT开发)和OFA(有效的嵌入初始化)的性能。

英文摘要:

In recent years, the development of pre-trained language models (PLMs) has gained momentum, showcasing their capacity to transcend linguistic barriers and facilitate knowledge transfer across diverse languages. However, this progress has predominantly bypassed the inclusion of very-low resource languages, creating a notable void in the multilingual landscape. This paper addresses this gap by introducing four tailored PLMs specifically finetuned for Angolan languages, employing a Multilingual Adaptive Fine-tuning (MAFT) approach. In this paper, we survey the role of informed embedding initialization and synthetic data in enhancing the performance of MAFT models in downstream tasks. We improve baseline over SOTA AfroXLMR-base (developed through MAFT) and OFA (an effective embedding initialization) by 12.3 and 3.8 points respectively.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑