本文提出 Linear Alignment 算法，通过散度约束下策略优化的新参数化和闭式解，在单次推理中实现 LLM 与人类偏好对齐，无需标注或训练。
摘要：
基于语言模型（LLM）的 AI 助手的成功取决于从人类反馈中进行强化学习（RLHF），以理解并对齐用户意图。然而，PPO 等传统对齐算法受到复杂标注和训练要求的阻碍。这种依赖限制了 RLHF 的适用性，并阻碍了为多样化人类偏好定制的专业助手的发展。在本工作中，我们提出了 Linear Alignment，这是一种新颖的算法，可在单个推理步骤中使语言模型与人类偏好对齐，消除了对数据标注和模型训练的依赖。Linear Alignment 结合了一种用于散度约束下策略优化的新参数化方法，能够以闭式方式提取最优策略，并有助于直接估计对齐后的响应。在通用和个性化偏好数据集上的大量实验表明，Linear Alignment 在多种场景下显著提升了 LLM 对齐的性能和效率。我们的代码和数据集发布于 https://github.com/Wizardcoast/Linear_Alignment.git。
英文摘要：
The success of AI assistants based on Language Models (LLMs) hinges on Reinforcement Learning from Human Feedback (RLHF) to comprehend and align with user intentions. However, traditional alignment algorithms, such as PPO, are hampered by complex annotation and training requirements. This reliance limits the applicability of RLHF and hinders the development of professional assistants tailored to diverse human preferences. In this work, we introduce \textit{Linear Alignment}, a novel algorithm that aligns language models with human preferences in one single inference step, eliminating the reliance on data annotation and model training. Linear alignment incorporates a new parameterization for policy optimization under divergence constraints, which enables the extraction of optimal policy in a closed-form manner and facilitates the direct estimation of the aligned response. Extensive experiments on both general and personalized preference datasets demonstrate that linear alignment significantly enhances the performance and efficiency of LLM alignment across diverse scenarios. Our code and dataset is published on \url{https://github.com/Wizardcoast/Linear_Alignment.git}.
Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes
Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale Department of Computer Science Cranberry-Lemon University Pittsburgh, PA 15213, USA（计算机科学系， Cranberry-Lemon 大学）；Ji Q. Ren & Yevgeny LeNet Department of Computational Neuroscience University of the Witwatersrand Joburg, South Africa（计算神经科学系， 威特沃特斯兰大学）
WildFeedback: Aligning LLMs With In-situ User Interactions And Feedback
Microsoft Corporation（微软公司）；Purdue University（普渡大学）；Texas A&M University（德克萨斯农工大学）；University of California San Diego（加州大学圣地亚哥分校）；University of Southern California（南加州大学）
Aligning Language Models Using Follow-up Likelihood as Reward Signal
National University of Singapore（新加坡国立大学）；Peking University（北京大学）；The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；Tencent AI Lab（腾讯AI实验室）；Shenzhen Research Institute of Big Data（深圳大数据研究院）；University of Science and Technology of China（中国科学技术大学）
Evaluating and Aligning CodeLLMs on Human Preference
Alibaba Group（阿里巴巴集团）；Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）；University of Chinese Academy of Sciences（中国科学院大学）；Shanghai Jiao Tong University（上海交通大学）
PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment
Harbin Institute of Technology（哈尔滨工业大学）；Beijing Normal University - Hong Kong Baptist University United International College（北京师范大学-香港浸会大学联合国际学院）；Qilu University of Technology（齐鲁工业大学）
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
School of Computer Science, Peking University（北京大学计算机科学系）；Center on Frontiers of Computing Studies, Peking University（北京大学前沿计算研究中心）；National Research and Engineering Center of Software Engineering, Peking University（北京大学软件工程研究中心）
Aligning LLMs through Multi-perspective User Preference Ranking-based Feedback for Programming Question Answering
School of Computer Science and Technology, University of Science and Technology of China（中国科学技术大学计算机科学与技术学院）；Hefei University of Technology（合肥工业大学）；School of Artificial Intelligence and Data Science, University of Science and Technology of China（中国科学技术大学人工智能与数据科学学院）；University of Toronto（多伦多大学）
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University（复旦大学计算机科学学院）；Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）；The University of Sydney（悉尼大学）；Nanyang Technological University（南洋理工大学）
提出自参考 AI 反馈框架，让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据，显著提升强化学习对齐效果。
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
School of Computer Science and Engineering（计算机科学与工程学院）；Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications（新一代人工智能技术及其交叉应用关键实验室）；Ministry of Education（教育部）
ERA-IT: Aligning Semantic Models with Revealed Economic Preference for Real-Time and Explainable Patent Valuation
School of Computing（计算学院）；Macquarie University（麦考瑞大学）；School of Computer Science（计算机科学学院）；University of Technology Sydney（悉尼技术大学）；School of Management（管理学院）；Zhejiang University（浙江大学）
