Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used at inference-time to estimate LLM responses' adherence to those desired behaviors. However, there is little work measuring how robust these reward models are to distribution shifts. In this work, we evaluate how reward model performance - measured via accuracy and calibration (i.e. alignment between accuracy and confidence) - is affected by distribution shift. We show novel calibration patterns and accuracy drops due to OOD prompts and responses, and that the reward model is more sensitive to shifts in responses than prompts. Additionally, we adapt an OOD detection technique commonly used in classification to the reward model setting to detect these distribution shifts in prompts and responses.
GRAM-R$^2$: Self-Training Generative Foundation Reward Models for Reward Reasoning
School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);Pattern Recognition Center, WeChat AI, Tencent Inc.(腾讯公司微信AI模式识别中心)
GRAM: A Generative Foundation Reward Model for Reward Generalization
School of Computer Science and Engineering, Northeastern University, Shenyang, China(东北大学计算机科学与工程学院);NiuTrans Research, Shenyang, China(NiuTrans研究);CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS, Beijing, China(中国科学院行为科学重点实验室);Meituan Inc.(美团公司)
Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
IIIS, Tsinghua University, Beijing, China(清华大学信息科学技术学院,北京,中国);Shanghai Qi Zhi Institute, Shanghai, China(上海启智研究院,上海,中国);Department of Computer Science, University of California, Los Angeles, California, USA(计算机科学系,加州大学洛杉矶分校,美国,加州)
The Essential Role of Causality in Foundation World Models for Embodied AI
Microsoft Research(微软研究院);Technical University of Munich(慕尼黑工业大学);KTH royal institute of technology(皇家理工学院);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)
Reward Auditor: Inference on Reward Modeling Suitability in Real-World Perturbed Scenarios
School of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);School of Statistics and Data Science, Shanghai University of International Business and Economics(上海国际商务经济学院统计与数据科学学院);Data Science and Analytics Thrust, The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州)数据科学与分析方向)
The Climb Carves Wisdom Deeper Than the Summit: On the Noisy Rewards in Learning to Reason
GSAI, Renmin University of China(清华大学);Large Language Model Department, Tencent(腾讯大语言模型部门);University of Macau(澳门大学);School of Computer Science, Wuhan University(武汉大学计算机学院)