arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.11751cs.CL
收藏

reWordBench:通过变换输入对奖励模型的鲁棒性进行基准测试与改进

reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs

FAIR at Meta(Meta 的 FAIR 部门);MIT(麻省理工学院)

Zhaofeng Wu, Michihiro Yasunaga, Andrew Cohen, Yoon Kim, Asli Celikyilmaz, Marjan Ghazvininejad

AI总结:

本文构建 reWordBench 基准,发现现有奖励模型对保持语义的输入变换高度脆弱,并提出通过训练模型为释义赋予相似分数来提升鲁棒性,同时改善对齐效果。

摘要:

奖励模型已成为现代自然语言处理中的核心组成部分,不仅作为可扩展的文本评估器,而且在许多对齐流程和推理时算法中也是不可或缺的组件。然而,尽管近期奖励模型在标准基准上的性能有所提升,但这可能部分源于过拟合效应,从而干扰对其真实能力的理解。在本工作中,我们审视了奖励模型的鲁棒性以及这种过拟合的程度。我们构建了 reWordBench,该基准以保持语义或排序不变的方式系统性地变换奖励模型的输入。我们表明,即使仅进行轻微的输入变换,最先进的奖励模型也会遭受显著的性能下降,有时甚至降至显著低于随机准确率的水平,这表明其具有脆弱性。为了提高奖励模型的鲁棒性,我们提出显式训练它们为释义赋予相似的分数,并发现该方法也能提升对其他不同类型变换的鲁棒性。例如,我们的鲁棒奖励模型在 RewardBench 的 Chat Hard 子集上将此类性能下降减少了约一半。此外,当用于对齐时,我们的鲁棒奖励模型表现出更好的效用,并能产生更高质量的输出,在与标准训练的奖励模型的对比中,在多达 59% 的实例中胜出。

英文摘要:

Reward models have become a staple in modern NLP, serving as not only a scalable text evaluator, but also an indispensable component in many alignment recipes and inference-time algorithms. However, while recent reward models increase performance on standard benchmarks, this may partly be due to overfitting effects, which would confound an understanding of their true capability. In this work, we scrutinize the robustness of reward models and the extent of such overfitting. We build **reWordBench**, which systematically transforms reward model inputs in meaning- or ranking-preserving ways. We show that state-of-the-art reward models suffer from substantial performance degradation even with minor input transformations, sometimes dropping to significantly below-random accuracy, suggesting brittleness. To improve reward model robustness, we propose to explicitly train them to assign similar scores to paraphrases, and find that this approach also improves robustness to other distinct kinds of transformations. For example, our robust reward model reduces such degradation by roughly half for the Chat Hard subset in RewardBench. Furthermore, when used in alignment, our robust reward models demonstrate better utility and lead to higher-quality outputs, winning in up to 59% of instances against a standardly trained RM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑