arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.08565cs.CL
收藏

通过代理微调语言模型

Tuning Language Models by Proxy

University of Washington(华盛顿大学);Allen Institute for AI(艾伦人工智能研究所)

Alisa Liu, Xiaochuang Han, Yizhong Wang, Yulia Tsvetkov, Yejin Choi, Noah A. Smith

AI总结:

本文提出代理微调(proxy-tuning),一种轻量级解码时算法,通过微调小模型并将其与未微调版本的预测差异应用于大模型,在不访问大模型参数的情况下实现高效定制,实验证明其能弥合 Llama2-70B 与其微调版 88% 的性能差距,并适用于 GPT-3.5 等黑盒模型。

摘要:

尽管大型预训练语言模型具有通用能力,但它们始终能从进一步适应中获益,以更好地实现期望的行为。然而,微调这些模型已变得越来越耗费资源,或者在模型权重为私有(不公开)时变得不可能。我们引入了代理微调(proxy-tuning),这是一种轻量级的解码时算法,它在黑盒语言模型(LM)之上运行,以达到与直接微调相同的目的,但仅访问其在输出词汇表上的预测,而不访问其参数。我们的方法先微调一个较小的语言模型,然后应用微调后的小模型与未微调的小模型之间的预测差异,将未微调的大模型的原始预测向微调方向偏移,同时保留大规模预训练的优势。在实验中,当我们使用仅有 7B 规模的代理模型对 Llama2-70B 应用代理微调时,在知识、推理和安全基准的评估中,我们能够弥合 Llama2-70B 与其真正微调的聊天版本之间 88% 的差距。随后,我们通过将其应用于代码领域的领域适应,以及问答和数学问题的特定任务微调,证明了代理微调的通用性。最后,我们展示了如何对一个真正的黑盒语言模型 GPT-3.5 进行代理微调以实现时间适应,增加其对近期事件的知识。我们的工作展示了通过解码时引导,使用微调后的小语言模型来高效定制大型、潜在专有的语言模型的前景。

英文摘要:

Despite the general capabilities of large pretrained language models, they consistently benefit from further adaptation to better achieve desired behaviors. However, tuning these models has become increasingly resource-intensive, or impossible when model weights are private. We introduce proxy-tuning, a lightweight decoding-time algorithm that operates on top of black-box LMs to achieve the same end as direct tuning, but by accessing only its predictions over the output vocabulary, not its parameters. Our method tunes a smaller LM, then applies the difference between the predictions of the small tuned and untuned LMs to shift the original predictions of the larger untuned model in the direction of tuning, while retaining the benefits of larger-scale pretraining. In experiments, when we apply proxy-tuning to Llama2-70B using proxies of only 7B size, we can close 88% of the gap between Llama2-70B and its truly-tuned chat version, when evaluated across knowledge, reasoning, and safety benchmarks. We then demonstrate the generality of proxy-tuning by applying it to domain adaptation on code, and task-specific finetuning on question-answering and math problems. Finally, we show how to proxy-tune a truly black-box LM, GPT-3.5, for temporal adaptation, increasing its knowledge about recent events. Our work demonstrates the promise of using small tuned LMs to efficiently customize large, potentially proprietary LMs through decoding-time guidance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑