Learning from human feedback via proxy reward modeling has been studied to align Large Language Models (LLMs) with human values. However, achieving reliable training through that proxy reward model (RM) is not a trivial problem, and its behavior remained as a black-box. In this paper, we study the role of proxy rewards in the LLM alignment via `reverse reward engineering' by composing interpretable features as a white-box reward function. We aim to replicate the ground truth (gold) reward signal by achieving a monotonic relationship between the proxy and gold reward signals after training the model using the proxy reward in reinforcement learning (RL). Our findings indicate that successfully emulating the gold reward requires generating responses that are relevant with enough length to open-ended questions, while also ensuring response consistency in closed-ended questions. Furthermore, resulting models optimizing our devised white-box reward show competitive performances with strong open-source RMs in alignment benchmarks. We highlight its potential usage as a simple but strong reward baseline for the LLM alignment, not requiring explicit human feedback dataset and RM training. Our code is available at https://github.com/naver-ai/rethinking-proxy-reward.
Spurious Rewards: Rethinking Training Signals in RLVR
University of Washington, Seattle, WA, USA（华盛顿大学）；Allen Institute for Artificial Intelligence, Seattle, WA, USA（人工智能研究院）；University of California, Berkeley, Berkeley, CA, USA（加州大学伯克利分校）
When a person is not satisfied with how a robot performs a task, they can intervene to correct it. Reward learning methods enable the robot to adapt its reward function online based on such human input, but they rely on handcrafted features. When the correction cannot be explained by these features, recent work in deep Inverse Reinforcement Learning (IRL) suggests that the robot could ask for task demonstrations and recover a reward defined over the raw state space. Our insight is that rather than implicitly learning about the missing feature(s) from demonstrations, the robot should instead as…
Rethinking Personalized Reward Modeling for LLMs under Preference Heterogeneity via Group-Debiased Federated Learning
Institute of Engineering Research, Korea University（韩国大学工程研究所）；Kim Jaechul Graduate School of AI, KAIST（韩国科学技术院金在哲人工智能研究生院）；Industrial Management Engineering, Korea University（韩国大学产业管理工程学院）
VRPO: Rethinking Value Modeling for Robust RL under Noisy Supervision in LLM Post-Training
College of Computer Science and Artificial Intelligence, Fudan University（复旦大学计算机科学与人工智能学院）；Honor Device Co., Ltd（荣耀设备有限公司）；Institute of Trustworthy Embodied AI, Fudan University（复旦大学可信具身人工智能研究院）；Shanghai Key Laboratory of Multimodal Embodied AI（上海多模态具身人工智能重点实验室）
Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function, where the user provides a textual prompt containing a few examples (few-shot) or a description (zero-shot) of the desired behavior. Our approach leverages this proxy reward function in an RL framework. Sp…
Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewards
The University of Michigan-Shanghai Jiao Tong University Joint Institute (UM-SJTU JI)（密歇根大学-上海交通大学联合学院）；Duke Kunshan University（昆山杜克大学）；NetEase Fuxi AI Lab（网易伏羲人工智能实验室）
University of Freiburg（弗莱堡大学）；Bosch Center for Artificial Intelligence（博世人工智能中心）
We introduce Synthetic Environments (SEs) and Reward Networks (RNs), represented by neural networks, as proxy environment models for training Reinforcement Learning (RL) agents. We show that an agent, after being trained exclusively on the SE, is able to solve the corresponding real environment. While an SE acts as a full proxy to a real environment by learning about its state dynamics and rewards, an RN is a partial proxy that learns to augment or replace rewards. We use bi-level optimization to evolve SEs and RNs: the inner loop trains the RL agent, and the outer loop trains the parameters o…
From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation
The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Nanyang Technological University（南洋理工大学）；Macquarie University（麦考瑞大学）
In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart's law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed "gold-standard" reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward…
No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
Zhongguancun Academy（中关村学院）；Academy of Mathematics and Systems Science, Chinese Academy of Sciences（中国科学院数学与系统科学研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Peking University（北京大学）；Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）
School of Operations Research and Information Engineering, Cornell University（康奈尔大学运营研究与信息工程学院）；Department of Industrial Systems Engineering and Management, National University of Singapore（新加坡国立大学工业系统工程与管理系）；SC Johnson College of Business, Cornell University（康奈尔大学SC约翰逊商学院）；Michael G. Foster School of Business, University of Washington（华盛顿大学迈克尔·G·福斯特商学院）
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Shanghai Jiao Tong University（上海交通大学）；BIMSA（北京雁栖湖应用数学研究院）；AIR, Tsinghua University（清华大学智能产业研究院）；HeliXon Limited（华深智药）
Many practical applications of reinforcement learning require agents to learn from sparse and delayed rewards. It challenges the ability of agents to attribute their actions to future outcomes. In this paper, we consider the problem formulation of episodic reinforcement learning with trajectory feedback. It refers to an extreme delay of reward signals, in which the agent can only obtain one reward signal at the end of each trajectory. A popular paradigm for this problem setting is learning with a designed auxiliary dense reward function, namely proxy reward, instead of sparse environmental sig…
TR-ICRL: Test-Time Rethinking for In-Context Reinforcement Learning
The Hong Kong Polytechnic University（香港理工大学）；Meituan（美团）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；Xi’an Jiaotong University（西安交通大学）；East China Normal University（华东师范大学）；Northeastern University（东北大学）
