arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2205.01068cs.CLcs.LG
收藏

OPT:开放式预训练Transformer语言模型

OPT: Open Pre-trained Transformer Language Models

Meta AI
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle O… 展开作者

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, Luke Zettlemoyer

摘要:

大语言模型通常经过数十万个计算日的训练,在零样本学习和少样本学习方面表现出了卓越的能力。考虑到它们的计算成本,这些模型很难在没有大量资本的情况下复制。对于通过API提供的少数几个模型,无法访问完整的模型权重,这使得它们难以研究。我们提出了开放式预训练Transformers(OPT),这是一套仅解码器预训练的transformers,参数范围从125 M到175 B,我们的目标是完全负责任地与感兴趣的研究人员分享。我们发现OPT-175 B与GPT-3相当,而仅需要1/7的碳足迹来开发。我们还发布了我们的日志,详细介绍了我们面临的基础设施挑战,以及用于测试所有已发布模型的代码。

英文摘要:

Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑