High-quality preference datasets are essential for training reward models that can effectively guide large language models (LLMs) in generating high-quality responses aligned with human preferences. As LLMs become stronger and better aligned, permissively licensed preference datasets, such as Open Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for reward modeling. Methods that distil preference data from proprietary LLMs such as GPT-4 have restrictions on commercial usage imposed by model providers. To improve upon both generated responses and attribute labeling quality, we release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0). Using a powerful internal base model trained on HelpSteer2, we are able to achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming currently listed open and proprietary models, as of June 12th, 2024. Notably, HelpSteer2 consists of only ten thousand response pairs, an order of magnitude fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly efficient for training reward models. Our extensive experiments demonstrate that reward models trained with HelpSteer2 are effective in aligning LLMs. In particular, we propose SteerLM 2.0, a model alignment approach that can effectively make use of the rich multi-attribute score predicted by our reward models. HelpSteer2 is available at https://huggingface.co/datasets/nvidia/HelpSteer2 and code is available at https://github.com/NVIDIA/NeMo-Aligner
RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Models
TMLR Group（TMLR小组）；Hong Kong Baptist University（香港 Baptist大学）；TCL Corporate Research (HK) Co Ltd（TCL企业研究（香港）有限公司）；Cooperative Medianet Innovation Center Shanghai Jiao Tong University（合作中位网创新中心上海交通大学）；Department of Mathematics Hong Kong Baptist University（香港 Baptist大学数学系）
HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent Hunyuan（腾讯文脉）；School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Toronto（多伦多大学）；NVIDIA（英伟达）；Princeton University（普林斯顿大学）；Salesforce Research（Salesforce研究）
Limited Preference Data? Learning Better Reward Model with Latent Space Synthesis
Department of Computer Sciences, University of Wisconsin-Madison（威斯康星大学麦迪逊分校计算机科学系）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算机科学与数据科学学院）
ACE-RL: Adaptive Constraint-Enhanced Reward for Long-form Generation Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Zhongguancun Academy, Beijing, China（中关村学院，北京，中国）；Tsinghua University（清华大学）；Wuhan AI Research（武汉人工智能研究所）
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）；China Mobile Communications Company Limited Research Institute（中国移动通信有限公司研究院）
GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning
LMU Munich（慕尼黑大学）；Technical University of Munich（慕尼黑工业大学）；Fudan University（复旦大学）；University Heidelberg（海德堡大学）；University of Oxford（牛津大学）；Munich Center for Machine Learning（慕尼黑机器学习中心）
Enhanced LLM Reasoning by Optimizing Reward Functions with Search-Driven Reinforcement Learning
School of Electrical and Computer Engineering, University of Oklahoma（俄克拉荷马大学电气与计算机工程学院）；Intelligent Neuromorphic and Quantum Understanding for Innovative Research and Engineering (INQUIRE) Laboratory, University of Oklahoma（智能神经形态与量子理解创新研究与工程实验室，俄克拉荷马大学）
Graph Reasoning Paradigm: Structured and Symbolic Reasoning with Topology-Aware Reinforcement Learning for Large Language Models
Harbin Institute of Technology（哈尔滨工业大学）；State Key Laboratory of Cognitive Intelligence（认知智能国家重点实验室）；Tianjin Normal University（天津师范大学）；Pengcheng Laboratory（鹏城实验室）
Sequential labeling is a fundamental NLP task, forming the backbone of many applications. Supervised learning of Seq2Seq models has shown great success on these problems. However, the training objectives are still significantly disconnected with the metrics and desiderata we care about in practice. For example, a practical sequence tagging application may want to optimize for a certain precision-recall trade-off (of the top-k predictions) which is quite different from the standard objective of maximizing the likelihood of the gold labeled sequence. Thus to bridge this gap, we propose GROOT -- …
Systematic Reward Gap Optimization for Mitigating VLM Hallucinations
School of Software, Beihang University（北京航空航天大学软件学院）；Shanghai Innovation Institute（上海创新研究院）；Shanghai AI Laboratory（上海人工智能实验室）；Tsinghua University（清华大学）
Learning suitable and well-performing dialogue behaviour in statistical spoken dialogue systems has been in the focus of research for many years. While most work which is based on reinforcement learning employs an objective measure like task success for modelling the reward signal, we use a reward based on user satisfaction estimation. We propose a novel estimator and show that it outperforms all previous estimators while learning temporal dependencies implicitly. Furthermore, we apply this novel user satisfaction estimation model live in simulated experiments where the satisfaction estimation…
