可控偏好优化：迈向可控的多目标对齐
Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
AI总结：
针对多目标对齐中的“对齐税”和现有单向技术灵活性差的问题，提出可控偏好优化（CPO），通过显式指定不同目标的偏好分数引导模型生成响应，有效减轻对齐税并提升多目标对齐性能。
摘要：
人工智能中的对齐追求模型响应与人类偏好及价值观之间的一致性。在实践中，人类偏好的多面性不可避免地引入了所谓的“对齐税”——一种妥协，即在一个目标（例如，无害性）上的对齐提升可能会削弱其他目标（例如，有用性）的性能。然而，现有的对齐技术大多是单向的，导致在各种目标之间的权衡次优且灵活性差。为了应对这一挑战，我们主张以显式偏好为 LLMs 提供基础的重要性。我们引入了可控偏好优化（CPO），它显式地为不同目标指定偏好分数，从而引导模型生成满足要求的响应。我们的实验分析表明，对齐后的模型能够提供匹配“3H”（有用性、诚实性、无害性）期望中的各种偏好的响应。此外，通过引入多样化的数据和对齐目标，我们在单一目标对齐上超越了基线方法，从而减轻了对齐税的影响，并在多目标对齐中实现了改进。
英文摘要：
Alignment in artificial intelligence pursues the consistency between model responses and human preferences as well as values. In practice, the multifaceted nature of human preferences inadvertently introduces what is known as the "alignment tax" -a compromise where enhancements in alignment within one objective (e.g.,harmlessness) can diminish performance in others (e.g.,helpfulness). However, existing alignment techniques are mostly unidirectional, leading to suboptimal trade-offs and poor flexibility over various objectives. To navigate this challenge, we argue the prominence of grounding LLMs with evident preferences. We introduce controllable preference optimization (CPO), which explicitly specifies preference scores for different objectives, thereby guiding the model to generate responses that meet the requirements. Our experimental analysis reveals that the aligned models can provide responses that match various preferences among the "3H" (helpfulness, honesty, harmlessness) desiderata. Furthermore, by introducing diverse data and alignment goals, we surpass baseline methods in aligning with single objectives, hence mitigating the impact of the alignment tax and achieving improvements in multi-objective alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
