Inference-Time Scaling for Generalist Reward Modeling
DeepSeek-AI；Dept. of Computer Sci. & Tech., Tsinghua University（计算机科学与技术系，清华大学）；Institute for AI Industry Research (AIR), Tsinghua University（人工智能产业研究院（AIR），清华大学）
Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale. Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that $\textit{proper learning methods could enable effective inference-time scalability}$. A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the $\textbf{inference-time scalability of generalist RM}$, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and potential for inference-time scaling. For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in $\textbf{DeepSeek-GRM}$ models. Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide voting process for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models are released at Hugging Face and ModelScope.
Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
Generalization in deep reinforcement learning over unseen environment variations usually requires policy learning over a large set of diverse training variations. We empirically observe that an agent trained on many variations (a generalist) tends to learn faster at the beginning, yet its performance plateaus at a less optimal level for a long time. In contrast, an agent trained only on a few variations (a specialist) can often achieve high returns under a limited computational budget. To have the best of both worlds, we propose a novel generalist-specialist training framework. Specifically, w…
AgentSkiller: Scaling Generalist Agent Intelligence through Semantically Integrated Cross-Domain Data Synthesis
AMU, Baidu Inc.（百度公司）；School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）；Singapore Management University（新加坡管理学院）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）
Advancing LLM Reasoning Generalists with Preference Trees
Tsinghua University（清华大学）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Northeastern University（东北大学）；Renmin University of China（中国人民大学）；BUPT（北京邮电大学）；Tencent（腾讯）
UltraMedical: Building Specialized Generalists in Biomedicine
Tsinghua University（清华大学）；University of Washington（华盛顿大学）；The First Affiliated Hospital of Nanchang University（南昌大学第一附属医院）；Shanghai Jiao Tong University（上海交通大学）
TSRBench: A Comprehensive Multi-task Multi-modal Time Series Reasoning Benchmark for Generalist Models
Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）；University of Illinois at Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校）；University of Maryland, College Park（马里兰大学学院公园分校）；University of California, San Diego（加州大学圣地亚哥分校）
GRAM-R$^2$: Self-Training Generative Foundation Reward Models for Reward Reasoning
School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）；Pattern Recognition Center, WeChat AI, Tencent Inc.（腾讯公司微信AI模式识别中心）
Generalist Foundation Models Are Not Clinical Enough for Hospital Operations
Courant Institute School of Mathematics, Computing, and Data Science（Courant学院数学、计算与数据科学学院）；New York University（纽约大学）；Department of Neurosurgery（神经外科部）；Global AI Frontier Lab（全球人工智能前沿实验室）；Electrical and Computer Engineering（电气与计算机工程）；Tandon School of Engineering（Tandon工程学院）；Grossman School of Medicine（Grossman医学院）；Department of Medicine（医学部）；Department of Computer Science（计算机科学部）；Department of Surgery（外科部）；Division of Applied AI Technologies（应用人工智能技术部）；Department of Population Health（人口健康部）；ETH Zurich（苏黎世联邦理工学院）
One Filters All: A Generalist Filter for State Estimation
School of Vehicle and Mobility, Tsinghua University（清华大学车辆与移动系统学院）；College of Engineering, Peking University（北京大学工程学院）；College of AI, Tsinghua University（清华大学人工智能学院）
Enabling Energy-Efficient Simultaneous Multi-Task Reinforcement Learning through Spiking Neural Networks with Active Dendrites for Bio-inspired Generalist Agents
eBRAIN Lab New York University (NYU) Abu Dhabi（eBRAIN实验室 新 York大学（NYU）阿布扎赫德）
A longstanding goal of the field of AI is a method for learning a highly capable, generalist agent from diverse experience. In the subfields of vision and language, this was largely achieved by scaling up transformer-based models and training them on large, diverse datasets. Motivated by this progress, we investigate whether the same strategy can be used to produce generalist reinforcement learning agents. Specifically, we show that a single transformer-based model - with a single set of weights - trained purely offline can play a suite of up to 46 Atari games simultaneously at close-to-human …
National Taiwan University（国立台湾大学）；RIKEN-AIP（理化学研究所人工智能研究中心）
This work provides a thorough study on how reward scaling can affect performance of deep reinforcement learning agents. In particular, we would like to answer the question that how does reward scaling affect non-saturating ReLU networks in RL? This question matters because ReLU is one of the most effective activation functions for deep learning models. We also propose an Adaptive Network Scaling framework to find a suitable scale of the rewards during learning for better performance. We conducted empirical studies to justify the solution.
Specifying reward functions for complex tasks like object manipulation or driving is challenging to do by hand. Reward learning seeks to address this by learning a reward model using human feedback on selected query policies. This shifts the burden of reward specification to the optimal design of the queries. We propose a theoretical framework for studying reward learning and the associated optimal experiment design problem. Our framework models rewards and policies as nonparametric functions belonging to subsets of Reproducing Kernel Hilbert Spaces (RKHSs). The learner receives (noisy) oracle…
Deep Reinforcement Learning (RL) has emerged as a powerful paradigm to solve a range of complex yet specific control tasks. Yet training generalist agents that can quickly adapt to new tasks remains an outstanding challenge. Recent advances in unsupervised RL have shown that pre-training RL agents with self-supervised intrinsic rewards can result in efficient adaptation. However, these algorithms have been hard to compare and develop due to the lack of a unified benchmark. To this end, we introduce the Unsupervised Reinforcement Learning Benchmark (URLB). URLB consists of two phases: reward-fr…
Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling
Huawei Noah’s Ark Lab, Montréal, Canada（华为诺亚实验室，加拿大蒙特利尔）；McGill University and Mila - Québec AI Institute（麦吉尔大学和魁北克人工智能研究所）；Huawei Noah’s Ark Lab, Beijing, China（华为诺亚实验室，中国北京）
