偏好协调器：面向大型语言模型的提示感知多目标对齐
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
AI总结：
针对大型语言模型多目标对齐依赖手动偏好权重导致用户负担重、训练效率低的问题，提出PRO框架，通过轻量级偏好适配器自动推断提示特定权重，理论分析与实验验证其优于现有方法。
摘要：
尽管大型语言模型（LLMs）在各类自然语言处理任务中展现出卓越能力，但在实际部署中，使这些模型在多个目标上与不同人类偏好对齐仍是重大挑战。现有多目标对齐方法依赖手动指定的偏好权重，这不仅让用户承担困难的偏好指定任务，还因探索无关偏好组合导致训练效率欠佳。为缓解这些问题，我们提出名为PRO（即偏好协调器，PReference Orchestrator）的新型框架，其包含轻量级偏好适配器，可在训练和部署阶段自动推断提示特定的偏好权重。具体而言，该适配器通过在多个奖励模型的归一化奖励分数上训练（针对偏好响应），自动学习每个提示的合适偏好权重，这些分数本质上反映了目标间的有效偏好平衡。此外，我们提供理论分析，证明在多目标对齐场景中，我们的提示感知偏好机制性能优于固定偏好权重。跨多任务的大量实验表明，我们的方法比现有多目标对齐方法更有效。
英文摘要：
While Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks, aligning these models with varying human preferences across multiple objectives remains a significant challenge in practical deployments. Existing multi-objective alignment methods rely on manually specified preference weights, which not only burden users with difficult preference specification tasks but also lead to suboptimal training efficiency due to exploration of irrelevant preference combinations. To alleviate these issues, we propose a novel framework named PRO, i.e., PReference Orchestrator, which features a lightweight preference adapter that automatically infers prompt-specific preference weights during both training and deployment phases. Specifically, the adapter automatically learns appropriate preference weights for each prompt by training on normalized reward scores from multiple reward models for preferred responses, which inherently reflect effective preference balances across objectives. Additionally, We provide theoretical analysis proving that our prompt-aware preference mechanism achieves superior performance compared to fixed preference weights in multi-objective alignment scenarios. Extensive experiments across multiple tasks demonstrate the effectiveness of our method over existing multi-objective alignment approaches.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
