arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2110.08207cs.LGcs.CL
收藏

多任务训练支持Zero-Shot任务泛化

Multitask Prompted Training Enables Zero-Shot Task Generalization

Hugging Face;Brown University(布朗大学);Snorkel AI;BigScience;KFUPM(法赫德国王石油与矿业大学);IRISA(法国信息与随机系统研究所);IMATAG;Hyperscience;SAP(思爱普);NTU(南洋理工大学);UCSD(加州大学圣地亚哥分校);SambaNova Systems;Walmart Labs(沃尔玛实验室);VU Amsterdam(阿姆斯特丹自由大学);University of Virginia(弗吉尼亚大学);ASUS(华硕);ZEALS;NYU(纽约大学);IBM Research(IBM研究院);UC Berkeley(加州大学伯克利分校);Parity;Inria(法国国家信息与自动化研究所);Naver Labs Europe(Naver欧洲实验室);BITS Pilani(博拉理工学院);University of Rome(罗马大学);Stanford(斯坦福大学);Charles River Analytics(查尔斯河分析公司);Booz Allen(博思艾伦);EleutherAI
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan De… 展开作者

Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, Alexander M. Rush

摘要:

大语言模型最近已经被证明在不同的任务集合上获得合理的zero-shot泛化(Brown等人,2020年)。已经假设这是语言模型的预训练中的内隐多任务学习的结果(Radford等人,2019年)。zero-shot泛化是否可以直接通过外显多任务学习来诱导?为了大规模地测试这个问题,我们开发了一个系统,可以轻松地将任何自然语言任务映射到人类可读的提示形式。我们转换了大量的监督数据集,每个数据集都有多个不同措辞的提示。这些提示的数据集允许对模型执行完全保留任务的能力进行基准测试。我们微调预训练的编码器-解码器模型(Raffel等人,2020; Lester等人,2021年)在这个多任务混合覆盖各种各样的任务。该模型在几个标准数据集上实现了强大的zero-shot性能,通常优于高达16倍大小的模型。此外,我们的方法在BIG-Bench基准测试的一部分任务上获得了很好的性能,比模型的性能高出6倍。所有经过训练的模型都可以在https://github.com/bigscience-workshop/t-zero上找到,所有提示都可以在https://github.com/bigscience-workshop/promptsource上找到。

英文摘要:

Large language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models' pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pretrained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero and all prompts are available at https://github.com/bigscience-workshop/promptsource.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑