从噪声轨迹到稳定梯度：面向大型推理模型对齐的偏差-方差优化偏好优化
From Noisy Traces to Stable Gradients: Bias-Variance Optimized Preference Optimization for Aligning Large Reasoning Models
AI总结：
针对大型推理模型偏好对齐中轨迹采样导致的梯度方差问题，提出BVPO方法，通过混合高方差轨迹估计器和低方差空轨迹估计器优化偏差-方差权衡，理论证明降低方差并收紧收敛界，实验在多个基准上显著提升对齐和推理性能。
摘要：
大型推理模型（LRMs）在生成最终答案之前会生成中间推理轨迹，从而在多步骤和数学任务上取得显著性能提升。然而，将LRMs与人类偏好对齐（这是模型部署的关键前提）仍未被充分探索。偏好对齐的统计正确目标要求对推理轨迹进行边缘化，但该计算在实践中难以处理。一种常见的变通方法是优化单个采样轨迹，这会因随机轨迹采样引入大量梯度方差。为应对这一挑战，我们从偏差-方差权衡的角度来审视LRMs的偏好优化，并提出偏差-方差优化偏好优化（BVPO）方法，这是一种简单、即插即用的方法，混合了两种梯度估计器：一种基于轨迹的高方差估计器和一种通过禁用推理轨迹生成而获得的低方差空轨迹估计器。我们的理论表明，对于任何非平凡混合，BVPO都能严格减少轨迹引起的方差，提供混合权重的闭式解以最小化相对于真实边际梯度的均方误差，并在标准光滑性和步长条件下，收紧随机梯度下降的经典收敛界。实验上，BVPO在AlpacaEval 2上比最佳基线最多提升7.8个百分点，在Arena-Hard上提升6.8个百分点。尽管仅在通用对话数据上训练，BVPO在六个数学推理基准的平均值上也将基础模型的推理性能最多提升4.0个百分点。这些结果揭示了轨迹采样方差是关键瓶颈，并表明直接优化偏差-方差权衡能带来更稳定的训练和更强的整体性能。
英文摘要：
Large reasoning models (LRMs) generate intermediate reasoning traces before producing final answers, yielding strong gains on multi-step and mathematical tasks. Yet aligning LRMs with human preferences, a crucial prerequisite for model deployment, remains underexplored. The statistically correct objective for preference alignment requires marginalizing over reasoning traces, but this computation is intractable in practice. A common workaround optimizes a single sampled trajectory, which introduces substantial gradient variance from stochastic trace sampling. To address this challenge, we frame preference optimization for LRMs through the lens of the bias--variance trade-off and propose Bias--Variance Optimized Preference Optimization (BVPO), a simple, drop-in method that mixes two gradient estimators: a high-variance trace-based estimator and a low-variance empty-trace estimator obtained by disabling reasoning trace generation. Our theory shows that BVPO strictly reduces trace-induced variance for any nontrivial mixture, provides a closed-form choice of the mixing weight that minimizes mean-squared error relative to the true marginal gradient, and under standard smoothness and step-size conditions, tightens classical convergence bounds for stochastic gradient descent. Empirically, BVPO improves alignment over the best baseline by up to 7.8 points on AlpacaEval~2 and 6.8 points on Arena-Hard. Despite being trained only on general conversational data, BVPO also boosts reasoning performance for base models by up to 4.0 points on the average of six math reasoning benchmarks. These results identify variance from trace sampling as a key bottleneck and demonstrate that directly optimizing the bias--variance trade-off yields more stable training and stronger overall performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
