Reward models trained on human preference data have been proven to effectively align Large Language Models (LLMs) with human intent within the framework of reinforcement learning from human feedback (RLHF). However, current reward models have limited generalization capabilities to unseen prompts and responses, which can lead to an unexpected phenomenon known as reward over-optimization, resulting in a decline in actual performance due to excessive optimization of rewards. While previous research has advocated for constraining policy optimization, our study introduces a novel approach to enhance the reward model's generalization ability against distribution shifts by regularizing the hidden states. Specifically, we retain the base model's language model head and incorporate a suite of text-generation losses to preserve the hidden states' text-generation capabilities, while concurrently learning a reward head behind the same hidden states. Our experimental results demonstrate that the introduced regularization technique markedly improves the accuracy of learned reward models across a variety of out-of-distribution (OOD) tasks and effectively alleviates the over-optimization issue in RLHF, offering a more reliable and robust preference learning paradigm.
Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning
INRIA(法国国家信息与自动化研究所);Cohere(Cohere公司);DeepMind(DeepMind公司);EHESS(法国高等社会科学研究院);ENS-PSL(巴黎高等师范学院-巴黎文理研究大学);CNRS(法国国家科学研究中心);Meta AI Research(Meta人工智能研究院)
Supervising the search process produces reliable and generalizable information-seeking agents
Department of Computer Science, University of Virginia, USA(弗吉尼亚大学计算机科学系);National Library of Medicine, National Institutes of Health, USA(美国国立卫生研究院国家医学图书馆);Department of Computer Science, University of Illinois Urbana–Champaign, USA(伊利诺伊大学厄巴纳-香槟分校计算机科学系);Medical Oncology, Dana–Farber Cancer Institute, USA(达纳-法伯癌症研究所医学肿瘤科);Surgery, University of Alabama at Birmingham, USA(阿拉巴马大学伯明翰分校外科系);Department of Neurology, Yale School of Medicine, USA(耶鲁医学院神经病学系)
Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoning
Shanghai AI Lab(上海人工智能实验室);Xi’an Jiaotong University(西安交通大学);The University of Hong Kong(香港大学);Peking University(北京大学);Hong Kong University of Science and Technology(香港科技大学)
FlowRL: Matching Reward Distributions for LLM Reasoning
Shanghai Jiao Tong University(上海交通大学);Shanghai AI Laboratory(上海人工智能实验室);Microsoft Research(微软研究院);Tsinghua University(清华大学);Peking University(北京大学);Renmin University of China(中国人民大学);Stanford University(斯坦福大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所)
DEER: Disentangled Mixture of Experts with Instance-Adaptive Routing for Generalizable Machine-Generated Text Detection
Faculty of Electronic and Information Engineering, Xi’an Jiaotong University(电子与信息工程学院,西安交通大学);China Mobile Group(中国移动集团);Queen Mary University of London(伦敦大学玛丽女王学院);City University of Hong Kong(香港城市大学)
TimeSRL: Generalizable Time-Series Behavioral Modeling via Semantic RL-Tuned LLMs -- A Case Study in Mental Health
Columbia University(哥伦比亚大学);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Yale University(耶鲁大学);University of California, Los Angeles(加州大学洛杉矶分校);Google(谷歌);University of Virginia(弗吉尼亚大学)
Stanford University(斯坦福大学);Boston University(波士顿大学);Columbia University(哥伦比亚大学);University of Toronto(多伦多大学);Institut Teknologi Bandung(Bandung 工程技术大学);Monash University Indonesia(墨尔本大学印尼分校);Capital One
FlightGPT: Towards Generalizable and Interpretable UAV Vision-and-Language Navigation with Vision-Language Models
School of Intelligent Systems Engineering, Sun Yat-Sen University(中山大学智能系统工程学院);DP Technology(DP技术公司);Beijing University Of Posts and Telecommunications(北京邮电大学);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);Tongji University(同济大学);School of Integrated Innovation, Chulalongkorn University(朱拉隆功大学创新学院)