Reward models are central to aligning large language models (LLMs) with human preferences. Yet most approaches rely on pointwise reward estimates that overlook the epistemic uncertainty in reward models arising from limited human feedback. Recent work suggests that quantifying this uncertainty can reduce the costs of human annotation via uncertainty-guided active learning and mitigate reward overoptimization in LLM post-training. However, uncertainty-aware reward models have so far been adopted without thorough comparison, leaving them poorly understood. This work introduces a unified framework, RewardUQ, to systematically evaluate uncertainty quantification for reward models. We compare common methods along standard metrics measuring accuracy and calibration, and we propose a new ranking strategy incorporating both dimensions for a simplified comparison. Our experimental results suggest that model size and initialization have the most meaningful impact on performance, and most prior work could have benefited from alternative design choices. To foster the development and evaluation of new methods and aid the deployment in downstream applications, we release our open-source framework as a Python package. Our code is available at https://github.com/lasgroup/rewarduq.
Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models
National University of Singapore(新加坡国立大学);Nanyang Technological University(南洋理工大学);University College London(伦敦大学学院);Yale University(耶鲁大学);The Hong Kong University of Science and Technology(香港科技大学);University of Washington(华盛顿大学)
提出Uncertainty of Thoughts (UoT)算法,通过不确定性感知模拟、基于不确定性的奖励和奖励传播机制,增强大型语言模型主动提问寻求信息的能力,在医疗诊断等任务中平均成功率提升38.1%。
OracleTSC: Oracle-Informed Reward Hurdle and Uncertainty Regularization for Traffic Signal Control
Department of Computer Science and Software Engineering(计算机科学与软件工程系);Auburn University(亚伯拉罕大学);Department of Computer Science(计算机科学系);University of Iowa(爱荷华大学);Department of Electrical and Computer Engineering(电气与计算机工程系);Clemson University(克莱姆森大学)
BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewards
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
InfoDensity: Rewarding Information-Dense Traces for Efficient Reasoning
Institute for Infocomm Research (I 2 R), A*STAR, Singapore(信息与通信研究机构(I 2 R),A*STAR,新加坡);Centre for Frontier AI Research (CFAR), A*STAR, Singapore(前沿人工智能研究中心(CFAR),A*STAR,新加坡)
From Passive Metric to Active Signal: The Evolving Role of Uncertainty Quantification in Large Language Models
Salesforce AI Research(Salesforce AI研究院);Intuit(Intuit公司);Vanderbilt University(范德比大学);University of California, Davis(加州大学戴维斯分校);Vanderbilt University Medical Center(范德比大学医学中心)
Uncertainty-Aware Reward-based Deep Reinforcement Learning for Intent Analysis of Social Media Information
Virginia Tech(弗吉尼亚理工大学);University of Oslo(奥斯陆大学);DEVCOM Army Research Laboratory(DEVCOM陆军研究实验室);University of Texas at Dallas(德克萨斯大学达拉斯分校);University of the District of Columbia(哥伦比亚特区大学)
Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Baichuan Inc.(百川公司);Tsinghua University(清华大学)
Balance Between Efficient and Effective Learning: Dense2Sparse Reward Shaping for Robot Manipulation with Environment Uncertainty
Institute of Intelligent Machines, Hefei Institute of Physical Science, CAS(中国科学院合肥物质科学研究院智能机械研究所);Institute of Plasma Physics, Hefei Institute of Physical Science, CAS(中国科学院合肥物质科学研究院等离子体物理研究所)
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
National University of Defense Technology(国防科技大学);State Key Laboratory of Complex & Critical Software Environment(复杂与关键软件环境国家重点实验室);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室)
Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
Department of Biostatistics, Yale University(耶鲁大学生物统计学系);Department of Biostatistics, Harvard University(哈佛大学生物统计学系);Department of Quantitative Methods, Purdue University(普渡大学定量方法系)
University at Buffalo, The State University of New York(纽约州立大学布法罗分校);University of Colorado Boulder(科罗拉多大学博尔德分校);University of Central Florida(中佛罗里达大学);Air Force Research Laboratory(空军研究实验室)