arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.10019cs.CLcs.AI
收藏

站在巨型冰冻语言模型的肩膀上

Standing on the Shoulders of Giant Frozen Language Models

AI21 Labs(AI21实验室)
Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwartz, Amnon Shashua, Kevin L… 展开作者

Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwartz, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham

摘要:

巨大的预训练语言模型(LM)已经在各种任务上表现出令人惊讶的良好zero-shot能力。这就产生了一个吸引人的愿景,即一个单一的、多功能的模型,在不同的应用程序中具有广泛的功能。然而,目前用于利用“冻结”LM的主要技术--即,保持其权重不变-仍然经常表现不佳的微调方法,这些方法以任务相关的方式修改这些权重。反过来,这些人会健忘,并损害多功能性,这表明性能和多功能性之间的权衡。本文的主要信息是,目前的冻结模型技术,如提示微调只是冰山一角,更强大的方法,利用冻结LM可以做同样的微调在具有挑战性的领域,而不牺牲底层模型的多功能性。为了证明这一点,我们介绍了三种利用冻结模型的新方法:依赖于输入的提示微调,冻结读者,递归LM,其中每一个都大大提高了当前的冻结模型的方法。事实上,我们的一些方法甚至在目前由后者主导的领域中优于微调方法。每种方法的计算成本都高于现有的冻结模型方法,但相对于单次通过巨大的冻结LM仍然可以忽略不计。这些方法中的每一种都是有意义的贡献,但通过将这些贡献放在一起,我们的目标是让读者相信一个更广泛的信息,超越任何给定方法的细节:冻结模型具有未开发的潜力,微调通常是不必要的。

英文摘要:

Huge pretrained language models (LMs) have demonstrated surprisingly good zero-shot capabilities on a wide variety of tasks. This gives rise to the appealing vision of a single, versatile model with a wide range of functionalities across disparate applications. However, current leading techniques for leveraging a "frozen" LM -- i.e., leaving its weights untouched -- still often underperform fine-tuning approaches which modify these weights in a task-dependent way. Those, in turn, suffer forgetfulness and compromise versatility, suggesting a tradeoff between performance and versatility. The main message of this paper is that current frozen-model techniques such as prompt tuning are only the tip of the iceberg, and more powerful methods for leveraging frozen LMs can do just as well as fine tuning in challenging domains without sacrificing the underlying model's versatility. To demonstrate this, we introduce three novel methods for leveraging frozen models: input-dependent prompt tuning, frozen readers, and recursive LMs, each of which vastly improves on current frozen-model approaches. Indeed, some of our methods even outperform fine-tuning approaches in domains currently dominated by the latter. The computational cost of each method is higher than that of existing frozen model methods, but still negligible relative to a single pass through a huge frozen LM. Each of these methods constitutes a meaningful contribution in its own right, but by presenting these contributions together we aim to convince the reader of a broader message that goes beyond the details of any given method: that frozen models have untapped potential and that fine-tuning is often unnecessary.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑