并非所有偏好都同等重要：面向推理模型的稳定性感知与梯度高效对齐
Not All Preferences Deserve Gradients: Understanding Gradient Utility in Offline Reasoning Alignment
AI总结：
SAGE通过动态框架提升推理模型对齐的稳定性与梯度效率，通过信噪比优化和稳定性感知评分函数实现更高效的训练。
摘要：
基于偏好的对齐对于训练大型推理模型至关重要；然而，标准方法如直接偏好优化（DPO）通常将所有偏好对视为同等重要，忽略了训练实例的演变效用。这种静态方法往往导致低效或不稳定优化，因为它会浪费计算在梯度微不足道的平凡对上，并受到接近不确定决策边界样本引入的噪声影响。面对这些挑战，我们提出了SAGE（稳定性感知梯度效率），一种动态框架，旨在通过最大化策略更新的信噪比来提高对齐的可靠性。具体而言，SAGE整合了一个粗粒度课程机制，该机制根据模型能力刷新候选池，并结合一个细粒度的稳定性感知评分函数，该函数优先选择信息丰富且自信的错误，同时过滤掉不稳定样本。在多个数学推理基准上的实验表明，SAGE显著加速了收敛速度，并优于静态基线，突显了在推理对齐中策略感知、稳定性意识的数据选择的重要性。
英文摘要：
Offline preference optimization aligns reasoning models from fixed chosen--rejected pairs, yet standard methods apply gradient updates from every pair regardless of its training value under the current policy. We argue that this uniform treatment is wasteful and potentially harmful. From the perspective of gradient utility, we show that a pair's contribution depends jointly on informativeness and stability. Pair utility drifts as the policy evolves, high-gradient samples can coincide with high-curvature regions, leading to noisy and destabilizing updates, and the most effective supervision comes from stable confident errors where the model is reliably wrong yet curvature remains low. These findings motivate SAGE (Stability-Aware Gradient Efficiency), which maintains difficulty-stratified candidate pools refreshed during training and selects pairs within each pool by a forward-pass signal-to-curvature score. Only pairs with high current utility receive gradient computation; the rest are excluded from backpropagation. On mathematical reasoning benchmarks across multiple model scales, SAGE outperforms full-data and size-matched baselines while producing substantially smoother optimization trajectories.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
