揭示多目标对齐中的跨目标干扰
Uncovering Cross-Objective Interference in Multi-Objective Alignment
AI总结：
本文提出CTWA方法，通过保持目标奖励与训练信号的正协方差来缓解多目标对齐中的跨目标干扰问题，并分析了干扰与模型几何属性的关系。
摘要：
我们研究了大型语言模型（LLMs）多目标对齐中的一种持续性失败模式：训练会提高部分目标的性能，而其他目标则会退化。我们将这种现象正式定义为跨目标干扰，并在经典标量化算法上进行了首次系统研究，显示干扰普遍存在且表现出强烈的模型依赖性。为了解释这一现象，我们推导出一个局部协方差定律，表明当一个目标的奖励与标量化分数表现出正协方差时，该目标在一阶上会得到改善。我们进一步将此分析扩展到现代对齐中使用的截断替代目标，证明在温和条件下，协方差定律仍然有效。基于此分析，我们提出了协方差目标加权适应（CTWA），这是一种即插即用的方法，通过保持目标奖励与训练信号之间的正协方差来有效缓解跨目标干扰。最后，我们用Polyak--Łojasiewicz条件补充这些局部改进条件，建立了非凸标量化优化达到全局收敛的条件，并阐明了跨目标干扰如何依赖于特定模型的几何属性。
英文摘要：
We study a persistent failure mode in multi-objective alignment for large language models (LLMs), in which scalarized training improves only some objectives while the others degrade. We formalize this phenomenon as cross-objective interference and, to our knowledge, conduct the first systematic study of scalarization algorithms for multi-objective LLM alignment. The study shows that interference is pervasive across algorithms yet strongly model-dependent. To understand how interference arises, we derive a local covariance law stating that an objective improves or degrades at first order according to the sign of the covariance between its reward and the scalarized score. We extend this law to the clipped surrogate objectives of modern reinforcement fine-tuning and show that it still holds under mild conditions. Building on this law, we propose COVariance-floor Enforced Reweighting (COVER), a one-sided controller that raises an objective's weight only when the covariance between its reward and the clipped advantage weight falls below a target. Through extensive experiments, we find that COVER can mitigate cross-objective interference while matching linear scalarization when objectives already co-improve. Finally, to explain why interference is model-dependent, we complement the local covariance law with a global convergence analysis. This analysis gives sufficient conditions for the non-convex scalarized objective to satisfy the Polyak--Łojasiewicz condition and relates interference to model geometry.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
