arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.15065cs.CL
收藏

推理时策略适配器(IPA):无需微调即可定制超大规模语言模型

Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning

Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Washington(华盛顿大学);University of Southern California(南加州大学)
Ximing Lu, Faeze Brahman, Peter West, Jaehun Jang, Khyathi Chandu, Abhilasha Ravichander, Lianhui Qin, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouh… 展开作者

Ximing Lu, Faeze Brahman, Peter West, Jaehun Jang, Khyathi Chandu, Abhilasha Ravichander, Lianhui Qin, Prithviraj Ammanabrolu, Liwei Jiang, Sahana Ramnath, Nouha Dziri, Jillian Fisher, Bill Yuchen Lin, Skyler Hallinan, Xiang Ren, Sean Welleck, Yejin Choi

AI总结:

本文提出推理时策略适配器 IPA,通过强化学习训练的轻量适配器在解码阶段引导大模型,无需微调即可在五项文本生成任务上显著提升性能。

摘要:

尽管超大规模语言模型已在多种语言任务上展现出卓越性能,但仅通过提示词对这些语言模型进行控制的程度往往有限。直接微调此类语言模型虽能有效实现定制,但可能极其昂贵(例如 GPT-3),甚至对更广泛的研究群体而言并不可行(例如 GPT-4)。我们提出推理时策略适配器(Inference-time Policy Adapters,IPA),它能够在不微调语言模型的情况下高效定制 GPT-3 等语言模型。IPA 在解码阶段通过一个轻量级策略适配器引导大型基础模型,该适配器使用强化学习进行训练,以优化任意用户目标。在毒性降低和词汇约束生成等五项具有挑战性的文本生成任务上,IPA 相比现成语言模型持续带来显著提升。它优于具有竞争力的基线方法,有时甚至包括昂贵的微调。特别是,使用 IPA 定制 GPT-2 可以胜过 GPT-3,而使用 IPA 定制 GPT-3 则能在 GPT-3 的基础上带来大幅性能提升(有时甚至超过 GPT-4)。这些令人鼓舞的结果凸显了 IPA 作为定制超大规模语言模型的轻量级替代方案的潜力。

英文摘要:

While extreme-scale language models have demonstrated exceptional performance on a variety of language tasks, the degree of control over these language models through pure prompting can often be limited. Directly fine-tuning such language models can be effective for tailoring them, but it can be either extremely costly (e.g., GPT-3) or not even feasible for the broader community (e.g., GPT-4). We propose Inference-time Policy Adapters (IPA), which efficiently tailors a language model such as GPT-3 without fine-tuning it. IPA guides a large base model during decoding time through a lightweight policy adapter trained to optimize an arbitrary user objective with reinforcement learning. On five challenging text generation tasks, such as toxicity reduction and lexically constrained generation, IPA consistently brings significant improvements over off-the-shelf language models. It outperforms competitive baseline methods, sometimes even including expensive fine-tuning. In particular, tailoring GPT-2 with IPA can outperform GPT-3, while tailoring GPT-3 with IPA brings a major performance boost over GPT-3 (and sometimes even over GPT-4). Our promising results highlight the potential of IPA as a lightweight alternative to tailoring extreme-scale language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑