arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.24040cs.LGcs.AIcs.CL
收藏

RewardUQ:一种用于不确定性感知奖励模型的统一框架

RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models

ETH Zurich(苏黎世联邦理工学院);ETH AI Center(ETH人工智能中心)

Daniel Yang, Samuel Stante, Florian Redhardt, Lena Libon, Parnian Kassraie, Ido Hakimi, Barna Pásztor, Andreas Krause

AI总结:

RewardUQ提出了一种统一框架,用于系统评估奖励模型的不确定性量化,通过比较常用方法和提出新排序策略,揭示模型大小和初始化对性能的影响,并开源框架促进新方法的发展和应用。

摘要:

奖励模型在对齐大语言模型(LLMs)与人类偏好方面起着核心作用。然而,大多数方法依赖于点估计奖励评估,忽略了由于人类反馈有限而产生的奖励模型中的先验不确定性。近期研究表明,量化这种不确定性可以通过不确定性引导的主动学习来减少人工标注的成本,并缓解LLM训练后的奖励过优化问题。然而,目前不确定性感知的奖励模型尚未经过彻底的比较,导致其理解不足。本文介绍了一种统一框架RewardUQ,系统地评估奖励模型的不确定性量化。我们比较了常用方法,沿标准的准确性和校准性度量标准进行评估,并提出了一种新的排序策略,结合两个维度以实现简化比较。我们的实验结果表明,模型大小和初始化对性能影响最大,大多数先前工作可能从替代的设计选择中受益。为了促进新方法的发展和评估,并协助在下游应用中的部署,我们发布我们的开源框架作为Python包。我们的代码可在https://github.com/lasgroup/rewarduq获取。

英文摘要:

Reward models are central to aligning large language models (LLMs) with human preferences. Yet most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback. Recent work suggests that quantifying this uncertainty can reduce the costs of human annotation via uncertainty-guided active learning and mitigate reward overoptimization in LLM post-training. However, uncertainty-aware reward models have so far been adopted without thorough comparison, leaving them poorly understood. This work introduces a unified framework, RewardUQ, to systematically evaluate uncertainty quantification for reward models. We compare common methods along standard metrics measuring accuracy and calibration, and we propose a new ranking strategy incorporating both dimensions for a simplified comparison. Our experimental results suggest that model size and initialization have the most meaningful impact on performance, and most prior work could have benefited from alternative design choices. To foster the development and evaluation of new methods and aid the deployment in downstream applications, we release our open-source framework as a Python package. Our code is available at https://github.com/lasgroup/rewarduq.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑