通过集成价值引导实现推理时语言模型对齐
Inference-Time Language Model Alignment via Integrated Value Guidance
AI总结:
本文提出IVG方法,利用隐式和显式价值函数在token和块级别引导解码,无需微调即可在推理时高效对齐大型语言模型,在多项任务及AlpacaEval 2.0基准上显著提升了模型对齐效果与胜率。
摘要:
大型语言模型通常通过微调来与人类偏好对齐,但调整大型模型计算量大且复杂。在这项工作中,我们引入了Integrated Value Guidance(IVG),这是一种使用隐式和显式价值函数分别在token和块级别引导语言模型解码的方法,在推理时高效地实现对齐。该方法避免了直接微调的复杂性,并优于传统方法。在实验中,我们证明了IVG在各种任务中的通用性。在受控情感生成和摘要任务中,使用基于gpt2的价值函数作为推理时引导,该方法显著提升了大型模型的对齐效果。此外,在更具挑战性的指令遵循基准AlpacaEval 2.0中,我们展示了经过特定调整的现成价值函数均大幅提升了大型模型相对于gpt-4-turbo的长度控制胜率(例如,在Tulu引导下,Mistral-7B-Instruct-v0.2从19.51%提升至26.51%,Mixtral-8x7B-Instruct-v0.1从25.58%提升至33.75%)。
英文摘要:
Large language models are typically fine-tuned to align with human preferences, but tuning large models is computationally intensive and complex. In this work, we introduce $\textit{Integrated Value Guidance}$ (IVG), a method that uses implicit and explicit value functions to guide language model decoding at token and chunk-level respectively, efficiently aligning large language models purely at inference time. This approach circumvents the complexities of direct fine-tuning and outperforms traditional methods. Empirically, we demonstrate the versatility of IVG across various tasks. In controlled sentiment generation and summarization tasks, our method significantly improves the alignment of large models using inference-time guidance from $\texttt{gpt2}$-based value functions. Moreover, in a more challenging instruction-following benchmark AlpacaEval 2.0, we show that both specifically tuned and off-the-shelf value functions greatly improve the length-controlled win rates of large models against $\texttt{gpt-4-turbo}$ (e.g., $19.51\% \rightarrow 26.51\%$ for $\texttt{Mistral-7B-Instruct-v0.2}$ and $25.58\% \rightarrow 33.75\%$ for $\texttt{Mixtral-8x7B-Instruct-v0.1}$ with Tulu guidance).
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。