arXiv 2204.075802023-10-12cs.CLcs.AI

mGPT：少样本学习者的多语言化

mGPT: Few-Shot Learners Go Multilingual

SaluteDevices；HSE University（高等经济大学）；University of Oslo（奥斯陆大学）；AIRI；NUST MISiS（国立科技研究大学“莫斯科国立钢铁合金学院”）；Institute of Linguistics RAS（俄罗斯科学院语言学研究所）

Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, Tatiana Shavrina

备注：: Accepted for publication at Transactions of the Association for Computational Linguistics (TACL) To be presented at the Conference on Empirical Methods in Natural Language Processing (EMNLP 2023)

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出mGPT，两个基于GPT架构的多语言模型（13亿和130亿参数），在60种语言上训练，性能与XGLM相当，覆盖更多低资源语言，并开源了代码和模型。

摘要：

近期研究表明，自回归语言模型可以通过零样本和少样本学习范式成功解决许多自然语言处理任务，这为使用预训练语言模型开辟了新的可能性。本文介绍了两个自回归类GPT模型，参数量分别为13亿和130亿，使用维基百科和Colossal Clean Crawled Corpus在来自25个语系的60种语言上进行训练。我们使用GPT-2源码和稀疏注意力机制复现了GPT-3架构；Deepspeed和Megatron框架使我们能够有效地并行化训练和推理步骤。由此产生的模型在性能上与Facebook最近发布的XGLM模型相当，覆盖更多语言，并增强了独联体国家和俄罗斯少数民族语言的低资源语言的自然语言处理可能性。我们详细阐述了架构设计选择的动机，全面描述了数据准备流程，并训练了五个小版本模型以选择最优的多语言分词策略。我们测量了模型在所有覆盖语言上的困惑度，并在广泛的多语言任务上进行了评估，包括分类、生成、序列标注和知识探测。模型使用零样本和少样本方法进行评估。此外，我们将分类任务与最先进的多语言模型XGLM进行了比较。源代码和mGPT XL模型已公开发布。

英文摘要：

Recent studies report that autoregressive language models can successfully solve many NLP tasks via zero- and few-shot learning paradigms, which opens up new possibilities for using the pre-trained language models. This paper introduces two autoregressive GPT-like models with 1.3 billion and 13 billion parameters trained on 60 languages from 25 language families using Wikipedia and Colossal Clean Crawled Corpus. We reproduce the GPT-3 architecture using GPT-2 sources and the sparse attention mechanism; Deepspeed and Megatron frameworks allow us to parallelize the training and inference steps effectively. The resulting models show performance on par with the recently released XGLM models by Facebook, covering more languages and enhancing NLP possibilities for low resource languages of CIS countries and Russian small nations. We detail the motivation for the choices of the architecture design, thoroughly describe the data preparation pipeline, and train five small versions of the model to choose the most optimal multilingual tokenization strategy. We measure the model perplexity in all covered languages and evaluate it on the wide spectre of multilingual tasks, including classification, generative, sequence labeling and knowledge probing. The models were evaluated with the zero-shot and few-shot methods. Furthermore, we compared the classification tasks with the state-of-the-art multilingual model XGLM. source code and the mGPT XL model are publicly released.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。