arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.18853cs.LG
收藏

解码时具有多目标的语言模型对齐

Decoding-Time Language Model Alignment with Multiple Objectives

IIIS, Tsinghua University(清华大学信息科学技术学院);University of Washington(华盛顿大学);Allen Institute for AI(艾伦人工智能研究所)

Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah A. Smith, Simon S. Du

AI总结:

针对语言模型多目标对齐难题,本文提出解码时算法MOD,通过所有基础模型预测的线性组合支持任意目标权重,并给出闭式解、最优性保证及显著实验提升。

摘要:

使语言模型(LM)与人类偏好对齐已成为一项关键追求,使这些模型能够更好地服务于多样化的用户需求。现有方法主要集中于针对单一奖励函数优化LM,限制了它们对各种目标的适应性。在这里,我们提出了 multi-objective decoding(MOD,多目标解码),这是一种解码时算法,对于不同目标上的任意给定权重,它根据所有基础模型预测的线性组合输出下一个token。我们利用一类 f-散度正则化对齐方法(例如 PPO、DPO 及其变体)中的一种共同形式,通过 Legendre 变换识别出一个闭式解,并推导出一种高效的解码策略。在理论上,我们说明了为什么即使在自然场景中现有方法也可能是次优的,并为我们的方法获得了最优性保证。实证结果证明了该算法的有效性。例如,与参数合并基线相比,在同等地朝3个目标优化时,MOD实现了12.8%的总体奖励提升。此外,我们在组合三个不同模型规模的完全微调LLM上对MOD进行了实验,每个模型针对不同目标,例如安全性、编程和一般用户偏好。与需要仔细策划数据集混合以实现全面改进的传统方法不同,我们可以使用MOD快速试验偏好权重,以找到模型的最佳组合。我们的最佳组合将 Toxigen 上的毒性降低到接近0%,并在其他三个指标(即 Codex@1、GSM-COT、BBH-COT)上实现了7.9–33.3%的提升。

英文摘要:

Aligning language models (LMs) to human preferences has emerged as a critical pursuit, enabling these models to better serve diverse user needs. Existing methods primarily focus on optimizing LMs for a single reward function, limiting their adaptability to varied objectives. Here, we propose $\textbf{multi-objective decoding (MOD)}$, a decoding-time algorithm that outputs the next token from a linear combination of predictions of all base models, for any given weightings over different objectives. We exploit a common form among a family of $f$-divergence regularized alignment approaches (such as PPO, DPO, and their variants) to identify a closed-form solution by Legendre transform, and derive an efficient decoding strategy. Theoretically, we show why existing approaches can be sub-optimal even in natural settings and obtain optimality guarantees for our method. Empirical results demonstrate the effectiveness of the algorithm. For example, compared to a parameter-merging baseline, MOD achieves 12.8% overall reward improvement when equally optimizing towards $3$ objectives. Moreover, we experiment with MOD on combining three fully-finetuned LLMs of different model sizes, each aimed at different objectives such as safety, coding, and general user preference. Unlike traditional methods that require careful curation of a mixture of datasets to achieve comprehensive improvement, we can quickly experiment with preference weightings using MOD to find the best combination of models. Our best combination reduces toxicity on Toxigen to nearly 0% and achieves 7.9--33.3% improvement across other three metrics ($\textit{i.e.}$, Codex@1, GSM-COT, BBH-COT).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑