通过系统消息泛化实现对数千种偏好的对齐
Aligning to Thousands of Preferences via System Message Generalization
AI总结：
提出通过系统消息泛化实现个性化LLM对齐的新范式，构建Multifaceted数据集训练Janus模型，在多个基准上超越现有模型，兼顾个性化与通用对齐。
摘要：
尽管人类天生具有多样化的价值观，但当前的大语言模型（LLM）对齐方法通常假设将LLM与公众的普遍偏好对齐是最优的。采用更个性化的LLM对齐方法的一个主要挑战是其缺乏可扩展性，因为这需要反复获取偏好数据，并为每个个体的偏好训练新的奖励模型和LLM。为了解决这些挑战，我们提出了一种新范式，即用户可以在系统消息中指定他们最看重的方面，从而引导LLM的生成行为，使其更好地与用户的意图对齐。然而，简单应用这种方法并非易事，因为LLM通常在统一的系统消息（例如“你是一个有用的助手”）上进行训练，这限制了它们泛化到各种未见过的系统消息的能力。为了提高这种泛化能力，我们创建了Multifaceted Collection，这是一个偏好数据集，包含192k种超越通用有用性和无害性的价值观组合，覆盖65k条用户指令。利用该数据集，我们训练了一个名为Janus的7B LLM，并通过添加各种反映用户偏好的未见系统消息，在5个基准（AlpacaEval 2.0、FLASK、Koala、MT-Bench和Self-Instruct）的921个提示上对其进行了测试。Janus对Mistral 7B Instruct v0.2、GPT-3.5 Turbo和GPT-4的平局+胜率分别为75.2%、72.4%和66.4%。出乎意料的是，在三个专注于响应有用性的基准（AlpacaEval 2.0、MT-Bench、Arena Hard Auto v0.1）上，Janus也分别以+4.0%、+0.1%、+3.0%的优势超越了LLaMA 3 8B Instruct，这突显了使用大量系统消息进行训练也可以增强与公众普遍偏好的对齐。我们的代码、数据集、基准和模型可在https://github.com/kaistAI/Janus获取。
英文摘要：
Although humans inherently have diverse values, current large language model (LLM) alignment methods often assume that aligning LLMs with the general public's preferences is optimal. A major challenge in adopting a more individualized approach to LLM alignment is its lack of scalability, as it involves repeatedly acquiring preference data and training new reward models and LLMs for each individual's preferences. To address these challenges, we propose a new paradigm where users specify what they value most within the system message, steering the LLM's generation behavior to better align with the user's intentions. However, a naive application of such an approach is non-trivial since LLMs are typically trained on a uniform system message (e.g., "You are a helpful assistant") which limits their ability to generalize to diverse, unseen system messages. To improve this generalization, we create the Multifaceted Collection, a preference dataset with 192k combinations of values beyond generic helpfulness and harmlessness, spanning 65k user instructions. Using this dataset, we train a 7B LLM called Janus and test it on 921 prompts from 5 benchmarks (AlpacaEval 2.0, FLASK, Koala, MT-Bench, and Self-Instruct) by adding various unseen system messages that reflect user preferences. Janus achieves tie+win rate of 75.2%, 72.4%, and 66.4% against Mistral 7B Instruct v0.2, GPT-3.5 Turbo, and GPT-4, respectively. Unexpectedly, on three benchmarks focused on response helpfulness (AlpacaEval 2.0, MT-Bench, Arena Hard Auto v0.1), Janus also outperforms LLaMA 3 8B Instruct by a +4.0%, +0.1%, +3.0% margin, underscoring that training with a vast array of system messages could also enhance alignment to the general public's preference as well. Our code, dataset, benchmark, and models are available at https://github.com/kaistAI/Janus.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
