Inferring reward functions from human behavior is at the center of value alignment - aligning AI objectives with what we, humans, actually want. But doing so relies on models of how humans behave given their objectives. After decades of research in cognitive science, neuroscience, and behavioral economics, obtaining accurate human models remains an open research topic. This begs the question: how accurate do these models need to be in order for the reward inference to be accurate? On the one hand, if small errors in the model can lead to catastrophic error in inference, the entire framework of reward learning seems ill-fated, as we will never have perfect models of human behavior. On the other hand, if as our models improve, we can have a guarantee that reward accuracy also improves, this would show the benefit of more work on the modeling side. We study this question both theoretically and empirically. We do show that it is unfortunately possible to construct small adversarial biases in behavior that lead to arbitrarily large errors in the inferred reward. However, and arguably more importantly, we are also able to identify reasonable assumptions under which the reward inference error can be bounded linearly in the error in the human model. Finally, we verify our theoretical insights in discrete and continuous control tasks with simulated and human data.
Statistical Inference for Misspecified Contextual Bandits
Department of Statistics, University of Wisconsin–Madison(威斯康星大学麦迪逊分校统计系);School of Data Science and Society, University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校数据科学与社会学院)
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));New Jersey Institute of Technology(新泽西理工学院);Institute of Computing Technology, CAS(中国科学院计算技术研究所)
Machine learning approaches to uncover the neural mechanisms of motivated behaviour: from ADHD to individual differences in effort and reward sensitivity
School of Computing(计算学院);Dublin City University(都柏林城市大学)
Learn What's Left, Not What's Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization
University of Florida(佛罗里达大学);UC San Diego(加州大学圣迭戈分校);Northeastern University(东北大学);Northwestern University(西北大学);Stanford University(斯坦福大学);Universität Innsbruck(因斯布鲁克大学)
PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models
Fudan University(复旦大学);Shanghai AI Laboratory(上海人工智能实验室);Soochow University(苏州大学);Stony Brook University(石溪大学);The Chinese University of Hong Kong(香港中文大学)
Genotype-Conditioned Molecular Generation via Evidence-Grounded Multi-Objective Latent Perturbation in Diffusion Models
Department of Computer Science and Engineering(计算机科学与工程系);University of Notre Dame(诺克斯大学);Department of Chemistry and Biochemistry(化学与生物化学系);Lucy Family Institute for Data & Society(数据与社会学院)
Spectral Evolution Search: Efficient Inference-Time Scaling for Reward-Aligned Image Generation
School of Data Science and Engineering, East China Normal University, Shanghai, China(数据科学与工程学院,东华大学,上海,中国);Alibaba Group, Hangzhou, China(阿里巴巴集团,杭州,中国)