提出 Chain of Hindsight 技术，将各种反馈转化为句子序列微调语言模型，使其基于反馈生成输出并自我纠错，在摘要和对话基准及人类评估中显著提升对齐人类偏好的效果。
摘要：
从人类偏好中学习对于语言模型匹配人类需求以及与人类和社会价值观对齐非常重要。先前的工作通过从人类反馈中学习以理解和遵循指令，取得了显著成功。然而，这些方法要么基于人类标注者偏爱的手动挑选的模型生成结果，导致数据利用效率低下且难以普遍应用，要么依赖强化学习，而强化学习常常受限于不完美的奖励函数并依赖于极具挑战性的优化过程。在这项工作中，我们提出了一种新技术，Chain of Hindsight，它易于优化，并且能够从任何形式的反馈中学习，无论其极性如何。我们的想法受到人类如何从以语言形式呈现的大量反馈中学习的启发。我们将所有类型的反馈转换为句子序列，然后用这些序列来微调模型，从而利用语言模型的语言理解能力。我们将模型条件化于一系列与反馈配对的模型生成结果上。通过这样做，模型被训练成基于反馈生成输出，同时学习识别并纠正负面属性或错误。将我们的方法应用于大型语言模型，我们观察到 Chain of Hindsight 在使语言模型与人类偏好对齐方面显著超越了先前的方法。我们报告了在摘要和对话基准上的显著改进，并且在人类评估中我们的方法被明显偏好。
英文摘要：
Learning from human preferences is important for language models to match human needs and to align with human and social values. Prior works have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them inefficient in terms of data utilization and challenging to apply in general, or they depend on reinforcement learning, which often suffers from imperfect reward functions and relies on extremely challenging optimizations. In this work, we propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model, allowing us to take advantage of the language comprehension capabilities of language models. We condition the model on a sequence of model generations paired with feedback. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors. Applying our method to large language models, we observed that Chain of Hindsight significantly surpasses previous methods in aligning language models with human preferences. We report significant improvements on summarization and dialogue benchmarks, with our approach markedly preferred in human evaluations.
Specifying rewards for reinforcement learned (RL) agents is challenging. Preference-based RL (PbRL) mitigates these challenges by inferring a reward from feedback over sets of trajectories. However, the effectiveness of PbRL is limited by the amount of feedback needed to reliably recover the structure of the target reward. We present the PRIor Over Rewards (PRIOR) framework, which incorporates priors about the structure of the reward function and the preference feedback into the reward learning process. Imposing these priors as soft constraints on the reward learning objective reduces the amou…
Reinforcement learning has seen wide success in finetuning large language models to better align with instructions via human feedback. The so-called algorithm, Reinforcement Learning with Human Feedback (RLHF) demonstrates impressive performance on the GPT series models. However, the underlying Reinforcement Learning (RL) algorithm is complex and requires an additional training pipeline for reward and value networks. In this paper, we consider an alternative approach: converting feedback to instruction by relabeling the original one and training the model for better alignment in a supervised m…
Hamburg University of Technology（汉堡工业大学）；University of Hamburg（汉堡大学）
This paper focuses on robotic reinforcement learning with sparse rewards for natural language goal representations. An open problem is the sample-inefficiency that stems from the compositionality of natural language, and from the grounding of language in sensory data and actions. We address these issues with three contributions. We first present a mechanism for hindsight instruction replay utilizing expert feedback. Second, we propose a seq2seq model to generate linguistic hindsight instructions. Finally, we present a novel class of language-focused learning tasks. We show that hindsight instr…
HISR: Hindsight Information Modulated Segmental Process Rewards For Multi-turn Agentic Reinforcement Learning
Aerospace Information Research Institute, Chinese Academy of Sciences（中国科学院航空航天信息研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent Hunyuan（腾讯文脉）；School of Computer Science and Technology, Tianjin University（天津大学计算机科学与技术学院）
Building open-ended agents that can autonomously discover a diversity of behaviours is one of the long-standing goals of artificial intelligence. This challenge can be studied in the framework of autotelic RL agents, i.e. agents that learn by selecting and pursuing their own goals, self-organizing a learning curriculum. Recent work identified language as a key dimension of autotelic learning, in particular because it enables abstract goal sampling and guidance from social peers for hindsight relabelling. Within this perspective, we study the following open scientific questions: What is the imp…
Hindsight-Privileged RLVR: Single-Rollout Token Credit Without Group Sampling
South China University of Technology（华南理工大学）；Tsinghua University（清华大学）；The Hong Kong University of Science and Technology（香港科技大学）；Huawei 2012 Laboratories（华为2012实验室）；Shenzhen Technology University（深圳技术大学）
TTIC（丰田技术研究所）；Microsoft Research（微软研究院）；University of Michigan（密歇根大学）
We consider a Markov decision process with deterministic state transition dynamics, adversarially generated rewards that change arbitrarily from round to round, and a bandit feedback model in which the decision maker only observes the rewards it receives. In this setting, we present a novel and efficient online decision making algorithm named MarcoPolo. Under mild assumptions on the structure of the transition dynamics, we prove that MarcoPolo enjoys a regret of O(T^(3/4)sqrt(log(T))) against the best deterministic policy in hindsight. Specifically, our analysis does not rely on the stringent …
SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning
Tsinghua University（清华大学）；Zhejiang University（浙江大学）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；Tongji University（同济大学）
University of Tuebingen（蒂宾根大学）；University of California San Diego（加利福尼亚大学圣迭戈分校）
The field of artificial intelligence (AI) alignment aims to investigate whether AI technologies align with human interests and values and function in a safe and ethical manner. AI alignment is particularly relevant for large language models (LLMs), which have the potential to exhibit unintended behavior due to their ability to learn and adapt in ways that are difficult to predict. In this paper, we discuss methodological challenges for the alignment problem specifically in the context of LLMs trained to summarize texts. In particular, we focus on methods for collecting reliable human feedback …
Online Learning for Multi-Layer Hierarchical Inference under Partial and Policy-Dependent Feedback
Department of Electrical and Computer Engineering（电气与计算机工程系）；The University of Texas at Austin（德克萨斯大学奥斯汀分校）；DEVCOM Army Research Laboratory（国防部陆军研究实验室）
The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models
University of Oxford（牛津大学）；University of Pennsylvania（宾夕法尼亚大学）；Bocconi University（博科尼大学）；ML Commons；Contextual AI；Factored AI；UCL（伦敦大学学院）；Cohere；MetaAI；New York University（纽约大学）；AWS AI Labs（亚马逊云科技AI实验室）；Meedan
Alignment-Aware Model Adaptation via Feedback-Guided Optimization
Department of Computer Science, University of British Columbia, Location, Canada（不列颠哥伦比亚大学计算机科学系）；Vector Institute of AI, Toronto, Canada（人工智能向量研究所）；Stony Brook University, New York, USA（石溪大学）
Constructive Large Language Models Alignment with Diverse Feedback
Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Feedback Indicators: The Alignment between Llama and a Teacher in Language Learning
Humboldt-Universität zu Berlin Department of Computer Science（柏林洪堡大学计算机科学系）；Humboldt-Universität zu Berlin Language Centre（柏林洪堡大学语言中心）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
Microsoft Corporation（微软公司）；Purdue University（普渡大学）；Texas A&M University（德克萨斯农工大学）；University of California San Diego（加州大学圣地亚哥分校）；University of Southern California（南加州大学）
We address online learning in complex auction settings, such as sponsored search auctions, where the value of the bidder is unknown to her, evolving in an arbitrary manner and observed only if the bidder wins an allocation. We leverage the structure of the utility of the bidder and the partial feedback that bidders typically receive in auctions, in order to provide algorithms with regret rates against the best fixed bid in hindsight, that are exponentially faster in convergence in terms of dependence on the action space, than what would have been derived by applying a generic bandit algorithm …
Learning optimal policies from sparse feedback is a known challenge in reinforcement learning. Hindsight Experience Replay (HER) is a multi-goal reinforcement learning algorithm that comes to solve such tasks. The algorithm treats every failure as a success for an alternative (virtual) goal that has been achieved in the episode and then generalizes from that virtual goal to real goals. HER has known flaws and is limited to relatively simple tasks. In this thesis, we present three algorithms based on the existing HER algorithm that improves its performances. First, we prioritize virtual goals f…
