Crosslingual Generalization through Multitask Finetuning
Hugging Face；EleutherAI；Google Research（谷歌研究院）；Booz Allen Hamilton（博思艾伦汉密尔顿）；Nanyang Technological University（南洋理工大学）；UC Berkeley（加州大学伯克利分校）；Brown University（布朗大学）；Yale University（耶鲁大学）；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；PSAU（萨塔姆·本·阿卜杜勒阿齐兹亲王大学）；University of Cambridge（剑桥大学）；KFUPM（法赫德国王石油与矿产大学）
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, X… 展开作者
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, Colin Raffel
备注：
9 main pages (119 with appendix), 16 figures and 11 tables
Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting, but so far explorations of MTF have focused on English data and models. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages that appear only in the pretraining corpus. Finetuning on multilingual tasks with English prompts further improves performance on English and non-English tasks leading to various state-of-the-art zero-shot results. We also investigate finetuning on multilingual tasks with prompts that have been machine-translated from English to match the language of each dataset. We find training on these machine-translated prompts leads to better performance on human-written prompts in the respective languages. Surprisingly, we find models are capable of zero-shot generalization to tasks in languages they have never intentionally seen. We conjecture that the models are learning higher-level capabilities that are both task- and language-agnostic. In addition, we introduce xP3, a composite of supervised datasets in 46 languages with English and machine-translated prompts. Our code, datasets and models are freely available at https://github.com/bigscience-workshop/xmtf.
Transformer based language models have led to impressive results across all domains in Natural Language Processing. Pretraining these models on language modeling tasks and finetuning them on downstream tasks such as Text Classification, Question Answering and Neural Machine Translation has consistently shown exemplary results. In this work, we propose a Multitask Finetuning methodology which combines the Bilingual Machine Translation task with an auxiliary Causal Language Modeling task to improve performance on the former task on Indian Languages. We conduct an empirical study on three languag…
LMU Munich（慕尼黑大学）；University of Cambridge（剑桥大学）；Monash University（莫纳什大学）；Technion, IIT（以色列理工学院）
Few-shot crosslingual transfer has been shown to outperform its zero-shot counterpart with pretrained encoders like multilingual BERT. Despite its growing popularity, little to no attention has been paid to standardizing and analyzing the design of few-shot experiments. In this work, we highlight a fundamental risk posed by this shortcoming, illustrating that the model exhibits a high degree of sensitivity to the selection of few shots. We conduct a large-scale experimental study on 40 sets of sampled few shots for six diverse NLP tasks across up to 40 languages. We provide an analysis of succ…
It has been shown for English that discrete and soft prompting perform strongly in few-shot learning with pretrained language models (PLMs). In this paper, we show that discrete and soft prompting perform better than finetuning in multilingual cases: Crosslingual transfer and in-language training of multilingual natural language inference. For example, with 48 English training examples, finetuning obtains 33.74% accuracy in crosslingual transfer, barely surpassing the majority baseline (33.33%). In contrast, discrete and soft prompting outperform finetuning, achieving 36.43% and 38.79%. We als…
BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting
Brown University（布朗大学）；EleutherAI；Yale University（耶鲁大学）；Hugging Face；MBZUAI（穆罕默德·本·扎耶德人工智能大学）；University College London（伦敦大学学院）；PSAU（沙特王子萨塔姆·本·阿卜杜勒阿齐兹大学）；Nanyang Technological University（南洋理工大学）；University of Washington（华盛顿大学）；University of Oregon（俄勒冈大学）；The Hong Kong University of Science and Technology（香港科技大学）；Booz Allen Hamilton（博思艾伦咨询公司）；NAVER LABS Europe（NAVER欧洲实验室）
We present a simple yet effective approach to build multilingual speech-to-text (ST) translation by efficient transfer learning from pretrained speech encoder and text decoder. Our key finding is that a minimalistic LNA (LayerNorm and Attention) finetuning can achieve zero-shot crosslingual and cross-modality transfer ability by only finetuning less than 10% of the pretrained parameters. This enables effectively leveraging large pretrained models with low training cost. Using wav2vec 2.0 for acoustic modeling, and mBART for multilingual text generation, our approach advanced the new state-of-t…
The Ohio State University（俄亥俄州立大学）；MIT-IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）；Massachusetts Institute of Technology（麻省理工学院）
Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks. However, existing methods typically learn soft prompt vectors from scratch, and it has not been clear how to exploit the rich cross-task knowledge with prompt vectors in a multitask learning setting. We propose multitask prompt tuning (MPT), which first learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn…
Hugging Face；Google（谷歌）；LightOn；Allen Institute for AI（艾伦人工智能研究所）；LPENS, École Normale Supérieure（巴黎高等师范学院物理实验室）
Large pretrained Transformer language models have been shown to exhibit zero-shot generalization, i.e. they can perform a wide variety of tasks that they were not explicitly trained on. However, the architectures and pretraining objectives used across state-of-the-art models differ significantly, and there has been limited systematic comparison of these factors. In this work, we present a large-scale evaluation of modeling choices and their impact on zero-shot generalization. In particular, we focus on text-to-text models and experiment with three model architectures (causal/non-causal decoder…
