For many tasks, the reward function is inaccessible to introspection or too complex to be specified procedurally, and must instead be learned from user data. Prior work has evaluated learned reward functions by evaluating policies optimized for the learned reward. However, this method cannot distinguish between the learned reward function failing to reflect user preferences and the policy optimization process failing to optimize the learned reward. Moreover, this method can only tell us about behavior in the evaluation environment, but the reward may incentivize very different behavior in even a slightly different deployment environment. To address these problems, we introduce the Equivalent-Policy Invariant Comparison (EPIC) distance to quantify the difference between two reward functions directly, without a policy optimization step. We prove EPIC is invariant on an equivalence class of reward functions that always induce the same optimal policy. Furthermore, we find EPIC can be efficiently approximated and is more robust than baselines to the choice of coverage distribution. Finally, we show that EPIC distance bounds the regret of optimal policies even under different transition dynamics, and we confirm empirically that it predicts policy training success. Our source code is available at https://github.com/HumanCompatibleAI/evaluating-rewards.
STARC: A General Framework For Quantifying Differences Between Reward Functions
Oxford University(牛津大学);University of Bristol(布里斯托尔大学);University of California, Berkeley(加州大学伯克利分校);FAR AI, Inc.(FAR AI公司);Future of Humanity Institute(人类未来研究所);Berkeley Artificial Intelligence Research(伯克利人工智能研究院);Department of Computer Science(计算机科学系);Bristol AI Safety Centre(布里斯托尔人工智能安全中心)
Explanation through Reward Model Reconciliation using POMDP Tree Search
Department of Aerospace Engineering Sciences, University of Colorado Boulder(科罗拉多大学博尔德分校航空航天工程科学系);Applied Physics Laboratory, Johns Hopkins University(约翰霍普金斯大学应用物理实验室)
Machine learning approaches to uncover the neural mechanisms of motivated behaviour: from ADHD to individual differences in effort and reward sensitivity
School of Computing(计算学院);Dublin City University(都柏林城市大学)
TDRM: Smooth Reward Models with Temporal Difference for LLM RL and Inference
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);University of Alberta(阿尔伯塔大学);California Institute of Technology(加州理工学院);School of Electronics and Computer Science, University of Southampton(南安普顿大学电子与计算机科学学院)
The Reward Model Selection Crisis in Personalized Alignment
School of Informatics, University of Edinburgh(信息学院,爱丁堡大学);Agency for Science, Technology and Research (A*STAR)(科技研究局);Samsung AI Research Center(三星人工智能研究中心)
Aligning Few-Step Diffusion Models with Dense Reward Difference Learning
School of Computer Science, National Engineering Research Center for Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering, Wuhan University(计算机学院、多媒体软件国家工程研究中心和多媒体与网络通信工程湖北省重点实验室、武汉大学);School of Cyber Science and Technology, Shenzhen Campus of Sun Yat-sen University(网络安全科学与技术学院、中山大学深圳校区);TikTok, ByteDance(TikTok、字节跳动);Tencent Inc.(腾讯公司);College of Electronic and Information Engineering, Tongji University(电子信息工程学院、同济大学);School of Medical Information and Engineering, Southwest Medical University(医学信息与工程学院、西南医科大学);College of Computing and Data Science and the Generative AI Lab at Nanyang Technological University(计算与数据科学学院和南洋理工大学生成式AI实验室)