arXiv 2211.032632022-11-23cs.CLcs.AIcs.LG

AfroLM：一种基于自主动学习、面向23种非洲语言的多语言预训练语言模型

AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages

Mila Quebec AI Institute（米拉魁北克人工智能研究所）；McGill University（麦吉尔大学）；Instituto Politécnico Nacional（国家理工学院）；Uppsala University（乌普萨拉大学）；Universidad de Deusto（德乌斯托大学）；Ashesi University（阿希西大学）；Montclair State University（蒙特克莱尔州立大学）；The College of Saint Rose（圣罗斯学院）；Technical University of Munich（慕尼黑工业大学）

Bonaventure F. P. Dossou, Atnafu Lambebo Tonja, Oreen Yousuf, Salomey Osei, Abigail Oppong, Iyanuoluwa Shode, Oluwabusayo Olufunke Awoyomi, Chris Chinenye Emezue

备注：: Third Workshop on Simple and Efficient Natural Language Processing, EMNLP 2022

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对非洲语言训练数据匮乏问题，提出基于自主动学习框架的多语言预训练模型AfroLM，在23种非洲语言上从零预训练，以小14倍的数据集超越AfriBERTa等基线，展现出强大的跨域泛化能力。

摘要：

近年来，多语言预训练语言模型因其在众多下游自然语言处理任务（NLP）上的出色表现而备受瞩目。然而，预训练这些大型多语言语言模型需要大量训练数据，而这在非洲语言中是缺乏的。主动学习是一种半监督学习算法，模型持续动态地学习识别最有益的样本进行训练，以在下游任务中实现更好的优化和性能。此外，主动学习能有效且实际地解决现实世界中的数据稀缺问题。尽管有诸多益处，主动学习在NLP尤其是多语言语言模型预训练的背景下却鲜受关注。本文提出AfroLM，这是一个使用我们新颖的自主动学习框架，从零开始在23种非洲语言上进行预训练的多语言语言模型（迄今为止最大规模的尝试）。在数据集规模比现有基线小得多（14倍）的情况下进行预训练，AfroLM在各种NLP下游任务（NER、文本分类和情感分析）上优于许多多语言预训练语言模型（AfriBERTa、XLMR-base、mBERT）。额外的域外情感分析实验表明，AfroLM能够跨各种领域良好泛化。我们在https://github.com/bonaventuredossou/MLM_AL发布了代码源及框架中使用的数据集。

英文摘要：

In recent years, multilingual pre-trained language models have gained prominence due to their remarkable performance on numerous downstream Natural Language Processing tasks (NLP). However, pre-training these large multilingual language models requires a lot of training data, which is not available for African Languages. Active learning is a semi-supervised learning algorithm, in which a model consistently and dynamically learns to identify the most beneficial samples to train itself on, in order to achieve better optimization and performance on downstream tasks. Furthermore, active learning effectively and practically addresses real-world data scarcity. Despite all its benefits, active learning, in the context of NLP and especially multilingual language models pretraining, has received little consideration. In this paper, we present AfroLM, a multilingual language model pretrained from scratch on 23 African languages (the largest effort to date) using our novel self-active learning framework. Pretrained on a dataset significantly (14x) smaller than existing baselines, AfroLM outperforms many multilingual pretrained language models (AfriBERTa, XLMR-base, mBERT) on various NLP downstream tasks (NER, text classification, and sentiment analysis). Additional out-of-domain sentiment analysis experiments show that \textbf{AfroLM} is able to generalize well across various domains. We release the code source, and our datasets used in our framework at https://github.com/bonaventuredossou/MLM_AL.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。