基于偏好向量的自适应有益有害对齐
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
AI总结：
本文提出偏好向量框架，通过模块化方法实现细粒度的用户可控偏好调整，提升大语言模型的有益性和无害性平衡。
摘要：
确保大型语言模型（LLMs）既有益又无害是关键挑战，过于严格的约束会导致过度拒绝，而宽松的模型则可能生成有害内容。现有方法如基于人类反馈的强化学习（RLHF）和直接偏好优化（DPO）试图平衡这些权衡，但存在性能冲突、可控性有限和扩展性差的问题。为了解决这些问题，我们提出了偏好向量，一种受任务算术启发的新型框架。我们不将多个偏好纳入单一目标进行优化，而是分别在个体偏好上训练模型，提取行为变化作为偏好向量，并在测试时动态合并。这种模块化方法使细粒度、用户可控的偏好调整成为可能，并且能够无缝集成新偏好而无需重新训练。实验表明，我们的偏好向量框架在不造成过度保守的情况下提高了有益性，允许对偏好权衡进行平滑控制，并支持可扩展的多偏好对齐。
英文摘要：
Ensuring that large language models (LLMs) are both helpful and harmless is a critical challenge, as overly strict constraints can lead to excessive refusals, while permissive models risk generating harmful content. Existing approaches, such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO), attempt to balance these trade-offs but suffer from performance conflicts, limited controllability, and poor extendability. To address these issues, we propose Preference Vector, a novel framework inspired by task arithmetic. Instead of optimizing multiple preferences within a single objective, we train separate models on individual preferences, extract behavior shifts as preference vectors, and dynamically merge them at test time. This modular approach enables fine-grained, user-controllable preference adjustments and facilitates seamless integration of new preferences without retraining. Experiments show that our proposed Preference Vector framework improves helpfulness without excessive conservatism, allows smooth control over preference trade-offs, and supports scalable multi-preference alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
