Aligning Large Language Models (LLMs) to human preferences in content, style, and presentation is challenging, in part because preferences are varied, context-dependent, and sometimes inherently ambiguous. While successful, Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) are prone to the issue of proxy reward overoptimization. Analysis of the DPO loss reveals a critical need for regularization for mislabeled or ambiguous preference pairs to avoid reward hacking. In this work, we develop a pessimistic framework for DPO by introducing preference uncertainty penalization schemes, inspired by offline reinforcement learning. The penalization serves as a correction to the loss which attenuates the loss gradient for uncertain samples. Evaluation of the methods is performed with GPT2 Medium on the Anthropic-HH dataset using a model ensemble to obtain uncertainty estimates, and shows improved overall performance compared to vanilla DPO, as well as better completions on prompts from high-uncertainty chosen/rejected responses.
Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles
National University of Defense Technology（国防科技大学）；State Key Laboratory of Complex & Critical Software Environment（复杂与关键软件环境国家重点实验室）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Peng Cheng Laboratory（鹏城实验室）
Contextual Online Uncertainty-Aware Preference Learning for Human Feedback
Department of Biostatistics, Harvard T.H. Chan School of Public Health（哈佛大学T.H.陈公共卫生学院生物统计学系）；Department of Biostatistics & Bioinformatics, Duke University（杜克大学生物统计学与生物信息学系）
TUR-DPO: Topology- and Uncertainty-Aware Direct Preference Optimization
Artificial Intelligence and Innovation Centre, University of Kurdistan, Erbil, Iraq（人工智能与创新中心，乌尔米耶大学，伊拉克）；Department of Computer Engineering, University of Kurdistan, Iran（计算机工程系，乌尔米耶大学，伊朗）；Centre for Artificial Intelligence Research and Optimisation, Torrens University Australia, Brisbane, Australia（人工智能研究与优化中心，塔伦斯大学澳大利亚，布里斯班，澳大利亚）；Research and Innovation Center, Obuda University, Budapest 1034, Hungary（研究与创新中心，奥布达大学，布达佩斯1034，匈牙利）；Center for Machine Vision and Signal Analysis (CMVS), University of Oulu, Finland（机器视觉与信号分析中心（CMVS），奥卢大学，芬兰）
When Modalities Conflict: How Unimodal Reasoning Uncertainty Governs Preference Dynamics in MLLMs
Peking University（北京大学）；Provable Responsible AI and Data Analytics (PRADA) Lab（可证明负责任的人工智能和数据分析实验室）；King Abdullah University of Science and Technology（卡布斯大学）；South China University of Technology（华南理工大学）；Tsinghua University（清华大学）；University of Georgia（佐治亚大学）；MBZUAI（马克斯·普朗克人工智能研究所）
On uncertainty-penalized Bayesian information criterion
Graduate School of Information Science and Technology（信息科学与技术研究生院）；Osaka University（大阪大学）；SANKEN (The Institute of Scientific and Industrial Research)（科学与工业研究所（SANKEN））
Adaptation of uncertainty-penalized Bayesian information criterion for parametric partial differential equation discovery
Graduate School of Information Science and Technology, Osaka University（大阪大学信息科学与技术研究生院）；SANKEN (The Institute of Scientific and Industrial Research), Osaka University（大阪大学产业科学研究所（SANKEN））
Trust Region Constrained Bayesian Optimization with Penalized Constraint Handling
Statistical Quality Control and Operations Research Unit, Indian Statistical Institute, Kolkata, India（印度统计研究院统计质量控制与运筹研究单位，加尔各答，印度）；Operations Management Group, Indian Institute of Management Calcutta, Kolkata, India（印度管理学院加尔各答运营管理组，加尔各答，印度）
Adaptive Preference Optimization with Uncertainty-aware Utility Anchor
State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（人工智能研究院，合肥综合性国家科学中心）；State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室，BIGAI）
Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Baichuan Inc.（百川公司）；Tsinghua University（清华大学）
Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
Department of Biostatistics, Yale University（耶鲁大学生物统计学系）；Department of Biostatistics, Harvard University（哈佛大学生物统计学系）；Department of Quantitative Methods, Purdue University（普渡大学定量方法系）
Multi-output Gaussian Processes for Uncertainty-aware Recommender Systems
Siemens AG（西门子股份公司）；German Cancer Consortium (DKTK)（德国癌症联盟（DKTK））；German Cancer Research Center (DKFZ)（德国癌症研究中心（DKFZ））；Goethe University Frankfurt（法兰克福歌德大学）
Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification
Department of Statistics and Data Science, UCLA（统计与数据科学系，加州大学洛杉矶分校）；Department of Statistics, University of Michigan（统计系，密歇根大学）；Department of Statistics and Data Science, and of Biostatistics, UCLA（统计与数据科学系及生物统计学系，加州大学洛杉矶分校）
