Reinforcement learning with AI feedback (RLAIF) is a popular paradigm for improving the instruction-following abilities of powerful pre-trained language models. RLAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL), using feedback from a critic model. While recent popular open-source models have demonstrated substantial improvements in performance from the RL step, in this paper we question whether the complexity of this RL step is truly warranted for AI feedback. We show that the improvements of the RL step are virtually entirely due to the widespread practice of using a weaker teacher model (e.g. GPT-3.5) for SFT data collection than the critic (e.g., GPT-4) used for AI feedback generation. Specifically, we show that simple supervised fine-tuning with GPT-4 as the teacher outperforms existing RLAIF pipelines. More generally, we find that the gains from RLAIF vary substantially across base model families, test-time evaluation protocols, and critic models. Finally, we provide a mechanistic explanation for when SFT may outperform the full two-step RLAIF pipeline as well as suggestions for making RLAIF maximally useful in practice.
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）；Nanyang Technological University（南洋理工大学）
提出自参考 AI 反馈框架，让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据，显著提升强化学习对齐效果。
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
Microsoft Corporation（微软公司）；Purdue University（普渡大学）；Texas A&M University（德克萨斯农工大学）；University of California San Diego（加州大学圣地亚哥分校）；University of Southern California（南加州大学）
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Hefei University of Technology（合肥工业大学）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）；University of Toronto（多伦多大学）
Training language models to follow instructions with human feedback
OpenAI（开放人工智能研究中心）；Anthropic（Anthropic公司）；Alignment Research Center（对齐研究中心）
使语言模型更大并不能使它们更好地遵循用户的意图。例如，大语言模型可能会生成不真实的、有害的或对用户毫无帮助的输出。换句话说，这些模型并不符合其用户。在本文中，我们展示了一种通过人工反馈进行微调，使语言模型与用户在各种任务上的意图保持一致的方法。从一组标注器编写的提示和通过OpenAI API提交的提示开始，我们收集了所需模型行为的标注器演示数据集，我们使用监督学习来微调GPT-3。然后，我们收集一个模型输出排名的数据集，我们使用它来进一步微调这个监督模型，使用基于人类反馈的强化学习。我们将生成的模型称为InstructGPT。在对我们的即时分布的人类评估中，尽管参数少了100倍，但1.3B参数InstructGPT模型的输出优于175 B GPT-3的输出。此外，InstructGPT模型在真实性方面有所改进，减少了有毒输出的生成，同时在公共NLP数据集上具有最小的性能回归。尽管InstructGPT仍然会犯一些简单的错误，但我们的研究结果表明，利用人类反馈进行微调是将语言模型与人类意图对齐的一个有希望的方向。
Aligning Language Models Using Follow-up Likelihood as Reward Signal
National University of Singapore（新加坡国立大学）；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Tencent AI Lab（腾讯AI实验室）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Science and Technology of China（中国科学技术大学）
Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
Meta；Institute for Clarity in Documentation（清晰文档研究所）；Inria Paris-Rocquencourt（Inria 巴黎-罗克quencourt 分所）；Rajiv Gandhi University（拉朱·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒尔研究实验室）
DiverValue-Bench: A Benchmark and Fine-Tuning Framework for Aligning Large Language Models with Diverse Human Values
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Renmin University of China（中国人民大学）；Beijing Institute of AI Safety and Governance（北京人工智能安全与治理研究院）；University of Chinese Academy of Sciences（中国科学院大学）
Constructive Large Language Models Alignment with Diverse Feedback
Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）
