回到Blackwell:在多目标偏好微调中闭合循环
Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuning
AI总结:
本文提出MaxEntBW和PROSPER算法,解决多目标偏好微调中的非传递性问题,通过直接处理多目标提升LLM微调效果。
摘要:
在偏好微调(PFT)中,处理非传递性(即循环)偏好是一个反复出现的挑战。非传递性偏好通常源于单一目标上的不一致排名或多个目标被缩放为单一度量。无论其来源如何,非传递性偏好的下游影响是相同的:没有明确的最优策略,这打破了标准PFT流程的核心假设。为此,我们提出了一种新的博弈论解决方案概念——最大熵Blackwell赢家(MaxEntBW),它在多目标非传递性偏好下是明确的。为了在大规模上计算MaxEntBW,我们推导出PROSPER:一个具有证明效率的PFT算法。与以往的自我扮演技术不同,PROSPER直接处理多个目标而不需要进行缩放。然后,我们将PROSPER应用于从多目标LLM-as-a-Judge反馈(例如基于评分标准的裁判)微调大型语言模型(LLMs)的问题,这是一个两种非传递性来源都出现的场景。我们发现,PROSPER在指令跟随和通用聊天基准测试中均优于所有考虑的基线,释放出7B和3B参数规模的训练模型检查点。
英文摘要:
A recurring challenge in preference fine-tuning (PFT) is handling $\textit{intransitive}$ (i.e., cyclic) preferences. Intransitive preferences often stem from either $\textit{(i)}$ inconsistent rankings along a single objective or $\textit{(ii)}$ scalarizing multiple objectives into a single metric. Regardless of their source, the downstream implication of intransitive preferences is the same: there is no well-defined optimal policy, breaking a core assumption of the standard PFT pipeline. In response, we propose a novel, game-theoretic solution concept -- the $\textit{Maximum Entropy Blackwell Winner}$ ($\textit{MaxEntBW}$) -- that is well-defined under multi-objective intransitive preferences. To enable computing MaxEntBWs at scale, we derive $\texttt{PROSPER}$: a provably efficient PFT algorithm. Unlike prior self-play techniques, $\texttt{PROSPER}$ directly handles multiple objectives without requiring scalarization. We then apply $\texttt{PROSPER}$ to the problem of fine-tuning large language models (LLMs) from multi-objective LLM-as-a-Judge feedback (e.g., rubric-based judges), a setting where both sources of intransitivity arise. We find that $\texttt{PROSPER}$ outperforms all baselines considered across both instruction following and general chat benchmarks, releasing trained model checkpoints at the 7B and 3B parameter scales.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。