ALaRM: Align Language Models via Hierarchical Rewards Modeling
Institute of Science and Technology for Brain-inspired Intelligence, Fudan University（复旦大学脑智能科学与技术研究院）；School of Data Science, Fudan University（复旦大学数据科学学院）；School of Computer Science, Fudan University（复旦大学计算机科学技术学院）；Research Institute of Intelligent and Complex Systems, Fudan University（复旦大学智能与复杂系统研究院）
We introduce ALaRM, the first framework modeling hierarchical rewards in reinforcement learning from human feedback (RLHF), which is designed to enhance the alignment of large language models (LLMs) with human preferences. The framework addresses the limitations of current alignment approaches, which often struggle with the inconsistency and sparsity of human supervision signals, by integrating holistic rewards with aspect-specific rewards. This integration enables more precise and consistent guidance of language models towards desired outcomes, particularly in complex and open text generation tasks. By employing a methodology that filters and combines multiple rewards based on their consistency, the framework provides a reliable mechanism for improving model alignment. We validate our approach through applications in long-form question answering and machine translation tasks, employing gpt-3.5-turbo for pairwise comparisons, and demonstrate improvements over existing baselines. Our work underscores the effectiveness of hierarchical rewards modeling in refining LLM training processes for better human preference alignment. We release our code at https://ALaRM-fdu.github.io.
Does LLM Alignment Really Need Diversity? An Empirical Study of Adapting RLVR Methods for Moral Reasoning
Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Microsoft Research（微软研究院）；University of Michigan（密歇根大学）；Shanghai Jiao Tong University（上海交通大学）；CUHKSZ（香港中文大学（深圳））；THU（清华大学）
Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
Chinese University of Hong Kong（香港中文大学）；Microsoft Research Asia（微软亚洲研究院）；Peking University（北京大学）；Hong Kong University of Science and Technology（香港科技大学）
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education（大数据与人工智能交通运输联合实验室，（北京交通大学）教育部）；School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China（计算机科学与技术学院，北京交通大学，北京，中国）；Tencent Inc, China（腾讯公司，中国）
Johannes Kepler University Linz（林茨约翰内斯·开普勒大学）；Dynatrace Research（Dynatrace研究院）；enliteAI（enliteAI公司）；Microsoft Research（微软研究院）；Institute of Advanced Research in Artificial Intelligence（高级人工智能研究所）
Reinforcement learning algorithms require many samples when solving complex hierarchical tasks with sparse and delayed rewards. For such complex tasks, the recently proposed RUDDER uses reward redistribution to leverage steps in the Q-function that are associated with accomplishing sub-tasks. However, often only few episodes with high rewards are available as demonstrations since current exploration strategies cannot discover them in reasonable time. In this work, we introduce Align-RUDDER, which utilizes a profile model for reward redistribution that is obtained from multiple sequence alignme…
Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
Institute of Computing and Intelligence, Harbin Institute of Technology, Shenzhen, China（计算与智能研究所，哈尔滨工业大学，深圳，中国）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室，深圳，中国）
One obstacle to applying reinforcement learning algorithms to real-world problems is the lack of suitable reward functions. Designing such reward functions is difficult in part because the user only has an implicit understanding of the task objective. This gives rise to the agent alignment problem: how do we create agents that behave in accordance with the user's intentions? We outline a high-level research direction to solve the agent alignment problem centered around reward modeling: learning a reward function from interaction with the user and optimizing the learned reward function with rei…
Stanford University（斯坦福大学）；University of Washington（华盛顿大学）
Modern multi-agent reinforcement learning frameworks rely on centralized training and reward shaping to perform well. However, centralized training and dense rewards are not readily available in the real world. Current multi-agent algorithms struggle to learn in the alternative setup of decentralized training or sparse rewards. To address these issues, we propose a self-supervised intrinsic reward ELIGN - expectation alignment - inspired by the self-organization principle in Zoology. Similar to how animals collaborate in a decentralized manner with those in their vicinity, agents trained with …
PRPO: Aligning Process Reward with Outcome Reward in Policy Optimization
Shanghai University（上海大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Jiaotong University（上海交通大学）；Artificial Intelligence Incubation and Innovation Institute, Fudan University（复旦大学人工智能孵化与创新研究院）；Shanghai Academy of AI for Science（上海人工智能科学研究院）
University of Bristol（布里斯托大学）；CortAIx, Thales（泰雷兹公司 CortAIx）
The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Usi…
One Adapts to Any: Meta Reward Modeling for Personalized LLM Alignment
The Hong Kong Polytechnic University（香港理工大学）；Huawei Technologies Ltd.（华为技术有限公司）；National University of Singapore（新加坡国立大学）；University of Science and Technology of China（中国科学技术大学）
