arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2103.06678cs.CL
收藏

阿拉伯语预训练语言模型中变量、规模和任务类型的相互作用

The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models

New York University Abu Dhabi(纽约大学阿布扎比分校);Carnegie Mellon University in Qatar(卡内基梅隆大学卡塔尔分校)

Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, Nizar Habash

摘要:

在本文中,我们探讨了阿拉伯语预训练语言模型的语言变体,数据大小和微调任务类型的影响。为此,我们在阿拉伯语的三种变体上构建了三个预训练的语言模型:现代标准阿拉伯语(MSA),方言阿拉伯语和古典阿拉伯语,以及第四个语言模型,该语言模型是在这三种语言的混合上进行预训练的。我们还研究了预训练数据大小的重要性,通过构建额外的模型,这些模型是在MSA变体的缩小集上进行预训练的。我们将不同的模型相互比较,并通过在跨越12个数据集的5个NLP任务上对它们进行微调,将它们与8个公开可用的模型进行比较。我们的研究结果表明,变量接近预训练数据微调数据比预训练数据的大小更重要。我们利用这一洞察力在定义一个优化的系统选择模型的研究任务。

英文摘要:

In this paper, we explore the effects of language variants, data sizes, and fine-tuning task types in Arabic pre-trained language models. To do so, we build three pre-trained language models across three variants of Arabic: Modern Standard Arabic (MSA), dialectal Arabic, and classical Arabic, in addition to a fourth language model which is pre-trained on a mix of the three. We also examine the importance of pre-training data size by building additional models that are pre-trained on a scaled-down set of the MSA variant. We compare our different models to each other, as well as to eight publicly available models by fine-tuning them on five NLP tasks spanning 12 datasets. Our results suggest that the variant proximity of pre-training data to fine-tuning data is more important than the pre-training data size. We exploit this insight in defining an optimized system selection model for the studied tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑