arXiv 2212.095352023-05-27cs.CLcs.AIcs.LG

BLOOM+1：为BLOOM增加语言支持以实现零样本提示

BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting

Brown University（布朗大学）；EleutherAI；Yale University（耶鲁大学）；Hugging Face；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University College London（伦敦大学学院）；PSAU（沙特王子萨塔姆·本·阿卜杜勒阿齐兹大学）；Nanyang Technological University（南洋理工大学）；University of Washington（华盛顿大学）；University of Oregon（俄勒冈大学）；The Hong Kong University of Science and Technology（香港科技大学）；Booz Allen Hamilton（博思艾伦咨询公司）；NAVER LABS Europe（NAVER欧洲实验室）

Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, … 展开作者

Zheng-Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Indra Winata, Stella Biderman, Edward Raff, Dragomir Radev, Vassilina Nikoulina

备注：: ACL 2023

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文研究将BLOOM适配到预训练未见的新语言，发现适配器微调优于继续预训练，且语言适配数据规模是决定零样本提示性能的主要因素，并证明将新语言纳入多任务微调混合数据是教会BLOOMZ新语言的最有效方法。

摘要：

BLOOM模型是一个大型公开的多语言语言模型，但其预训练仅限于46种语言。为了在不产生极其高昂成本的情况下将BLOOM的优势扩展到其他语言，有必要将BLOOM适配到预训练期间未见的新语言。在这项工作中，我们将现有的语言适配策略应用于BLOOM，并在资源受限的环境下对其在八种新语言上的零样本提示性能进行了基准测试。我们发现语言适配能够有效提升新语言上的零样本性能。令人惊讶的是，我们发现对于大型模型而言，基于适配器的微调比继续预训练更有效。此外，我们发现提示性能并未受到语言特性（如书写系统）的显著影响，它主要由语言适配数据的规模决定。我们还将新语言添加到BLOOMZ中，BLOOMZ是BLOOM的一个多任务微调版本，能够零样本地遵循任务指令。我们发现将一种新语言纳入多任务微调混合数据中是教会BLOOMZ掌握新语言的最有效方法。我们得出结论，只要有足够的训练数据，语言适配可以很好地泛化到多样化的语言。我们的代码可在https://github.com/bigscience-workshop/multilingual-modeling获取。

英文摘要：

The BLOOM model is a large publicly available multilingual language model, but its pretraining was limited to 46 languages. To extend the benefits of BLOOM to other languages without incurring prohibitively large costs, it is desirable to adapt BLOOM to new languages not seen during pretraining. In this work, we apply existing language adaptation strategies to BLOOM and benchmark its zero-shot prompting performance on eight new languages in a resource-constrained setting. We find language adaptation to be effective at improving zero-shot performance in new languages. Surprisingly, we find that adapter-based finetuning is more effective than continued pretraining for large models. In addition, we discover that prompting performance is not significantly affected by language specifics, such as the writing system. It is primarily determined by the size of the language adaptation data. We also add new languages to BLOOMZ, which is a multitask finetuned version of BLOOM capable of following task instructions zero-shot. We find including a new language in the multitask fine-tuning mixture to be the most effective method to teach BLOOMZ a new language. We conclude that with sufficient training data language adaptation can generalize well to diverse languages. Our code is available at https://github.com/bigscience-workshop/multilingual-modeling.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。