通过原则引导解码实现即时偏好对齐
On-the-fly Preference Alignment via Principle-Guided Decoding
AI总结：
本文提出OPAD框架，在推理阶段通过原则引导奖励直接调整模型输出，无需微调即可实现通用与个性化偏好对齐，并取得优于或可比现有基线的效果。
摘要：
随着大型语言模型领域的迅速扩展，使模型生成内容与人类价值观和偏好保持一致正变得日益重要。诸如基于人类反馈的强化学习等流行对齐方法，在以更强控制力引导模型方面已显示出显著成功。然而，这些方法需要大量计算资源，因而效率低下；同时还需要收集大量训练数据，以适应人类偏好的多样性和多元性，这并不现实。这些限制显著制约了任务特定和通用偏好对齐方法的适用范围与效果。在本工作中，我们提出通过原则引导解码实现即时偏好对齐（On-the-fly Preference Alignment via Principle-Guided Decoding，OPAD），在推理期间直接使模型输出与人类偏好对齐，从而无需微调。我们的方法首先为原本不可行的优化问题构造一个替代解，然后基于该替代解设计原则引导的奖励函数。最终的对齐策略通过最大化这一定制化奖励得到，该奖励利用了受约束策略与其无约束对应策略之间的差异。OPAD在推理期间直接修改模型预测，确保遵循原则，同时不会产生重新训练或微调的计算开销。实验表明，OPAD在通用和个性化对齐任务中均取得具有竞争力或更优的性能，证明了其相较于最先进基线方法的效率和有效性。
英文摘要：
With the rapidly expanding landscape of large language models, aligning model generations with human values and preferences is becoming increasingly important. Popular alignment methods, such as Reinforcement Learning from Human Feedback, have shown significant success in guiding models with greater control. However, these methods require considerable computational resources, which is inefficient, and substantial collection of training data to accommodate the diverse and pluralistic nature of human preferences, which is impractical. These limitations significantly constrain the scope and efficacy of both task-specific and general preference alignment methods. In this work, we introduce On-the-fly Preference Alignment via Principle-Guided Decoding (OPAD) to directly align model outputs with human preferences during inference, eliminating the need for fine-tuning. Our approach involves first curating a surrogate solution to an otherwise infeasible optimization problem and then designing a principle-guided reward function based on this surrogate. The final aligned policy is derived by maximizing this customized reward, which exploits the discrepancy between the constrained policy and its unconstrained counterpart. OPAD directly modifies the model's predictions during inference, ensuring principle adherence without incurring the computational overhead of retraining or fine-tuning. Experiments show that OPAD achieves competitive or superior performance in both general and personalized alignment tasks, demonstrating its efficiency and effectiveness compared to state-of-the-art baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
