arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.09043cs.LGcs.CL
收藏

DynaCF: 通过动态反事实敏感性缓解奖励模型中的捷径学习

DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity

The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));New Jersey Institute of Technology(新泽西理工学院);Institute of Computing Technology, CAS(中国科学院计算技术研究所)

Fengyuan Liu, Yongliang Miao, Zirui He, Yanguang Liu, Fei Sun, Mengnan Du

AI总结:

提出DynaCF框架,通过在线测量反事实扰动下的边际变化和偏好翻转来动态降低捷径敏感样本的权重,从而缓解奖励模型中的捷径学习问题。

摘要:

从成对偏好中训练的奖励模型往往利用表面的捷径线索而非学习真正的响应质量。我们提出DynaCF,一个用于缓解奖励模型训练中捷径学习的动态重加权框架。与静态捷径启发式方法不同,DynaCF在优化过程中通过应用保持语义的反事实扰动并跟踪当前模型下产生的边际变化和偏好翻转,在线测量捷径敏感性。在Bradley-Terry目标中,具有较高捷径敏感性的样本被动态降低权重,鼓励模型较少依赖表面模式,更多依赖任务相关的偏好信号。大量实验表明,DynaCF在偏好建模中持续提高了鲁棒性。

英文摘要:

Reward models trained from pairwise preferences often exploit superficial shortcut cues rather than learning true response quality. We propose DynaCF, a dynamic reweighting framework for mitigating shortcut learning in reward model training. Unlike static shortcut heuristics, DynaCF measures shortcut sensitivity online during optimization by applying semantics-preserving counterfactual perturbations and tracking the resulting margin shifts and preference flips under the current model. Samples with higher shortcut sensitivity are dynamically downweighted in the Bradley-Terry objective, encouraging the model to rely less on superficial patterns and more on task-relevant preference signals. Extensive experiments show that DynaCF consistently improves robustness in preference modeling.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑