通过偏好维度扩展解释并打破安全与有益性天花板
Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansion
AI总结：
本文提出MORA方法，通过多维奖励整合解决多目标对齐中的安全与有益性矛盾，实验显示在序列对齐中提升5%-12.4%，同时对齐中整体奖励提升4.6%。
摘要：
在大型语言模型的多目标对齐领域，平衡不同的人类偏好常表现为零和冲突。内在竞争目标之间的紧张关系决定了在优化一个指标（如有益性）时，往往会对另一个指标（如无害性）造成显著惩罚。尽管先前工作主要关注数据选择、参数融合或训练中的算法平衡，但这些方法仅在固定帕累托前沿上强制妥协，未能根本解决内在的权衡问题。本文从多维奖励的新视角出发，通过扩展模型的 rollout 并分析不同奖励维度的输出，得出关键结论：多个目标之间的冲突源于提示本身限制了可实现的多维奖励。基于这一核心观察，我们提出MORA：多目标奖励整合。具体而言，MORA通过预采样隔离单奖励提示，并通过重写原始问题以融入多维意图来扩展奖励多样性。大量实验表明：（1）在序列对齐中，MORA在多次跨有益、无害和真实维度的对齐后，单偏好改进范围为5%至12.4%，尤其在无害性方面有显著提升。（2）在同时对齐中，MORA实现了平均整体奖励提升4.6%。我们的代码可在https://github.com/Shiying-Huang/MORA-MPA获取。
英文摘要：
In the realm of multi-objective alignment for large language models, balancing disparate human preferences often manifests as a zero-sum conflict. Specifically, the intrinsic tension between competing goals dictates that aggressively optimizing for one metric (e.g., helpfulness) frequently incurs a substantial penalty on another (e.g., harmlessness). While prior work mainly focuses on data selection, parameter merging, or algorithmic balancing during training, these approaches merely force compromises between divergent preferences along a fixed Pareto frontier, failing to fundamentally resolve the inherent trade-off. In this work, we approach this problem from a novel perspective of multi-dimensional rewards. By scaling up the model's rollouts and analyzing the outputs across different reward dimensions, we arrive at a critical conclusion: the conflict among multiple objectives stems from the fact that the prompt itself inherently restricts the achievable multi-dimensional rewards. Based on this core observation, we propose MORA: Multi-Objective Reward Assimilation. Specifically, MORA isolates single-reward prompts through pre-sampling and expands their reward diversity by rewriting the original questions to incorporate multi-dimensional intents. Extensive experiments demonstrate that: (1) in sequential alignment, MORA achieves single-preference improvements ranging from 5% to 12.4%, with exceptional gains in harmlessness, after multiple-preference alignment across helpful, harmless, and truthful dimensions. (2) In simultaneous alignment, MORA achieves an average overall reward improvement of 4.6%. Our codes are available at https://github.com/Shiying-Huang/MORA-MPA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
