Panacea：通过偏好适应实现LLM的帕累托对齐
Panacea: Pareto Alignment via Preference Adaptation for LLMs
AI总结：
本文提出Panacea方法，通过基于SVD的低秩适应将偏好向量在线注入，实现单一LLM以帕累托最优方式适应多样人类偏好，无需额外微调。
摘要：
当前的大语言模型对齐方法通常使用标量人类偏好标签。然而，这种惯例往往过度简化了人类偏好的多维性和异质性，导致表达力下降甚至出现对齐偏差。本文提出了Panacea，一种创新方法，将对齐重新构建为多维偏好优化问题。Panacea训练一个单一模型，能够在线且以帕累托最优的方式适应多样化的偏好集合，无需进一步微调。这里的一个主要挑战是使用低维偏好向量来引导模型行为，尽管模型由极其大量的参数所控制。为解决这一问题，Panacea被设计为使用基于奇异值分解（SVD）的低秩适应，使得偏好向量可以简单地在线作为奇异值注入。理论上，我们证明了在温和条件下，Panacea能够通过常见的损失聚合方法恢复整个帕累托前沿。此外，我们的实验首次证明了通过多种优化方法，将单一LLM对齐到指数级广泛的人类偏好谱系的可行性。我们的工作标志着在可控且帕累托最优的方式下，有效且高效地将模型对齐到多样且复杂的人类偏好方面迈出了一步。
英文摘要：
Current methods for large language model alignment typically use scalar human preference labels. However, this convention tends to oversimplify the multi-dimensional and heterogeneous nature of human preferences, leading to reduced expressivity and even misalignment. This paper presents Panacea, an innovative approach that reframes alignment as a multi-dimensional preference optimization problem. Panacea trains a single model capable of adapting online and Pareto-optimally to diverse sets of preferences without the need for further tuning. A major challenge here is using a low-dimensional preference vector to guide the model's behavior, despite it being governed by an overwhelmingly large number of parameters. To address this, Panacea is designed to use singular value decomposition (SVD)-based low-rank adaptation, which allows the preference vector to be simply injected online as singular values. Theoretically, we prove that Panacea recovers the entire Pareto front with common loss aggregation methods under mild conditions. Moreover, our experiments demonstrate, for the first time, the feasibility of aligning a single LLM to represent an exponentially vast spectrum of human preferences through various optimization methods. Our work marks a step forward in effectively and efficiently aligning models to diverse and intricate human preferences in a controllable and Pareto-optimal manner.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
