Reward models (RMs) are crucial for the training of large language models (LLMs), yet they typically rely on large-scale human-annotated preference pairs. With the widespread deployment of LLMs, in-the-wild interactions have emerged as a rich source of implicit reward signals. This raises the question: Can we develop reward models directly from in-the-wild interactions? In this work, we explore this possibility by adopting WildChat as an interaction source and proposing a pipeline to extract reliable human feedback, yielding 186k high-quality instances for training WildReward via ordinal regression directly on user feedback without preference pairs. Extensive experiments demonstrate that WildReward achieves comparable or even superior performance compared to conventional reward models, with improved calibration and cross-sample consistency. We also observe that WildReward benefits directly from user diversity, where more users yield stronger reward models. Finally, we apply WildReward to online DPO training and observe significant improvements across various tasks. Code and data are released at https://github.com/THU-KEG/WildReward.
检测成功的行为对于训练智能代理至关重要。因此,可推广的奖励模型是能够学习推广其行为的代理的先决条件。在这项工作中,我们专注于开发强大的成功检测器,利用大型的预训练视觉语言模型(Flamingo,Alayrac et al.(2022))和人类奖励注释。具体地说,我们把成功检测作为一个视觉问答(VQA)问题,表示为VQA。我们研究了三个截然不同的领域的成功检测:(i)在模拟家庭中的交互式语言条件代理,(ii)真实世界的机器人操作,(iii)“在野外”的人类自我中心的视频。我们研究了基于Flamingo的成功检测模型在前两个领域中看不见的语言和视觉变化的泛化特性,并发现所提出的方法能够在分布测试场景中优于定制的奖励模型。在最后一个领域的“在野外”的人类视频,我们表明,成功检测看不见的真实视频提出了一个更具挑战性的推广任务,以确保未来的工作。我们希望我们的初步结果鼓励在现实世界中的成功检测和奖励建模的进一步工作。
Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation
University of Cambridge(剑桥大学);Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院);Field AI;University of Washington(华盛顿大学);University of Tennessee(田纳西大学);Korean Advanced Institute of Science and Technology(韩国高等科学技术院);University of California San Diego(加州大学圣地亚哥分校);The Alan Turing Institute(艾伦·图灵研究所);ServiceNow