通过交互使大语言模型与个体偏好对齐
Aligning LLMs with Individual Preferences via Interaction
AI总结：
针对大语言模型忽视个体偏好定制的问题，提出通过多轮对话隐式推断用户偏好并动态对齐的“交互式对齐”方法，构建3,310个用户画像和3K+对话数据集，经微调与强化学习训练，在ALOE基准上验证了有效性。
摘要：
随着大语言模型（LLMs）展现出日益强大的能力，使其行为与人类价值观和偏好对齐对其广泛应用变得至关重要。虽然以往的研究侧重于与有益性、无害性和诚实性等原则的一般性对齐，但考虑个体化和多样化偏好的需求在很大程度上被忽视，这可能损害定制化的人类体验。为解决这一差距，我们训练能够“交互式对齐”的大语言模型，本质上是培养大语言模型的元技能，即通过多轮对话隐式推断当前用户未言明的个性化偏好，然后动态地将其后续行为和响应与这些推断出的偏好对齐。我们的方法包括通过最初创建种子示例来建立一个包含3,310个不同用户画像的多样化池，然后通过迭代的自我生成和过滤进行扩展。在独特用户画像的引导下，我们利用多LLM协作开发了一个多轮偏好数据集，其中包含3K+个树结构的多轮对话。最后，我们应用监督微调和强化学习，利用该数据集增强大语言模型。为了评估，我们建立了ALOE（与定制化偏好对齐）基准，包含100个精心挑选的示例和设计良好的指标，以衡量对话过程中的定制化对齐性能。实验结果表明，我们的方法在通过交互实现动态、个性化对齐方面是有效的。
英文摘要：
As large language models (LLMs) demonstrate increasingly advanced capabilities, aligning their behaviors with human values and preferences becomes crucial for their wide adoption. While previous research focuses on general alignment to principles such as helpfulness, harmlessness, and honesty, the need to account for individual and diverse preferences has been largely overlooked, potentially undermining customized human experiences. To address this gap, we train LLMs that can ''interact to align'', essentially cultivating the meta-skill of LLMs to implicitly infer the unspoken personalized preferences of the current user through multi-turn conversations, and then dynamically align their following behaviors and responses to these inferred preferences. Our approach involves establishing a diverse pool of 3,310 distinct user personas by initially creating seed examples, which are then expanded through iterative self-generation and filtering. Guided by distinct user personas, we leverage multi-LLM collaboration to develop a multi-turn preference dataset containing 3K+ multi-turn conversations in tree structures. Finally, we apply supervised fine-tuning and reinforcement learning to enhance LLMs using this dataset. For evaluation, we establish the ALOE (ALign With CustOmized PrEferences) benchmark, consisting of 100 carefully selected examples and well-designed metrics to measure the customized alignment performance during conversations. Experimental results demonstrate the effectiveness of our method in enabling dynamic, personalized alignment via interaction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
