arXiv 2402.185712024-03-06cs.LGcs.AIcs.CLstat.ML

面向多样化用户偏好的大语言模型算术控制：多目标奖励下的方向偏好对齐

Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards

The Hong Kong University of Science and Technology（香港科技大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, Tong Zhang

备注：: The code and model are released at https://github.com/Haoxiang-Wang/directional-preference-alignment

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对标量奖励RLHF难以捕捉多样用户偏好的问题，提出多目标奖励下的方向偏好对齐框架DPA，将偏好建模为奖励空间方向，结合偏好条件化RSF微调，实现LLM生成的算术式细粒度控制，在Mistral-7B上验证了有效性。

摘要：

对大语言模型（LLMs）的细粒度控制仍是一项重大挑战，阻碍了其适配多样化的用户需求。尽管基于人类反馈的强化学习（RLHF）在LLM对齐方面展现出潜力，但其对标量奖励的依赖往往限制了其在实际应用中捕捉多样化用户偏好的能力。为解决这一局限，我们提出方向偏好对齐（Directional Preference Alignment, DPA）框架。与标量奖励RLHF不同，DPA引入多目标奖励建模来表征多样化的偏好分布。此外，DPA将用户偏好建模为奖励空间中的方向（即单位向量），以实现依赖用户的偏好控制。我们的方法包括训练一个多目标奖励模型，随后采用Llama 2所使用的RLHF方法——拒绝采样微调（Rejection Sampling Finetuning, RSF）的偏好条件化变体对LLM进行微调，该方法在各类奖励目标间实现了更优的性能权衡。与标量奖励RLHF相比，DPA为用户提供了对LLM生成的直观控制：用户可以通过算术方式指定期望的权衡（例如，更高的有用性、更低的冗余度）。我们还通过在Mistral-7B上开展的实际对齐实验验证了DPA的有效性。我们的方法可对有用性与冗余度之间的权衡实现直接的算术控制，同时保持与Direct Preference Optimization（DPO）等强基线相当的竞争力。

英文摘要：

Fine-grained control over large language models (LLMs) remains a significant challenge, hindering their adaptability to diverse user needs. While Reinforcement Learning from Human Feedback (RLHF) shows promise in aligning LLMs, its reliance on scalar rewards often limits its ability to capture diverse user preferences in real-world applications. To address this limitation, we introduce the Directional Preference Alignment (DPA) framework. Unlike the scalar-reward RLHF, DPA incorporates multi-objective reward modeling to represent diverse preference profiles. Additionally, DPA models user preferences as directions (i.e., unit vectors) in the reward space to achieve user-dependent preference control. Our method involves training a multi-objective reward model and then fine-tuning the LLM with a preference-conditioned variant of Rejection Sampling Finetuning (RSF), an RLHF method adopted by Llama 2. This method enjoys a better performance trade-off across various reward objectives. In comparison with the scalar-reward RLHF, DPA offers users intuitive control over LLM generation: they can arithmetically specify their desired trade-offs (e.g., more helpfulness with less verbosity). We also validate the effectiveness of DPA with real-world alignment experiments on Mistral-7B. Our method provides straightforward arithmetic control over the trade-off between helpfulness and verbosity while maintaining competitive performance with strong baselines such as Direct Preference Optimization (DPO).

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。