arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.08829cs.CLcs.AI
收藏

WildReward: 从真实世界的人类交互中学习奖励模型

WildReward: Learning Reward Models from In-the-Wild Human Interactions

Department of Computer Science and Technology, Tsinghua University(计算机科学与技术系,清华大学)

Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Lei Hou, Juanzi Li

AI总结:

本文提出WildReward,通过真实世界用户交互直接学习奖励模型,无需偏好对,实现更优的校准和一致性,并在多种任务中取得显著提升。

摘要:

奖励模型(RMs)对于大语言模型(LLMs)的训练至关重要,但通常依赖于大规模的人类标注偏好对。随着LLMs的广泛应用,在真实世界中的交互已涌现出丰富的隐含奖励信号。这引发了一个问题:能否直接从真实世界交互中开发奖励模型?在本文中,我们通过采用WildChat作为交互来源,并提出一个提取可靠人类反馈的流程,生成了186,000个高质量实例,通过顺序回归直接在用户反馈上训练WildReward,而无需偏好对。广泛的实验表明,WildReward在性能上与传统奖励模型相当或更优,具有改进的校准和跨样本一致性。我们还发现,WildReward直接受益于用户多样性,更多用户会产生更强的奖励模型。最后,我们将WildReward应用于在线DPO训练,并在各种任务上观察到显著改进。代码和数据已发布在https://github.com/THU-KEG/WildReward。

英文摘要:

Reward models (RMs) are crucial for the training of large language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deployment of LLMs, in-the-wild interactions have emerged as a rich source of implicit reward signals. This raises the question: Can we develop reward models directly from in-the-wild interactions? In this work, we explore this possibility by adopting WildChat as an interaction source and proposing a pipeline to extract reliable human feedback, yielding 186k high-quality instances for training WildReward via ordinal regression directly on user feedback without preference pairs. Extensive experiments demonstrate that WildReward achieves comparable or even superior performance compared to conventional reward models, with improved calibration and cross-sample consistency. We also observe that WildReward benefits directly from user diversity, where more users yield stronger reward models. Finally, we apply WildReward to online DPO training and observe significant improvements across various tasks. Code and data are released at https://github.com/THU-KEG/WildReward.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑