Large Language Models (LLMs) have become increasingly popular due to their ability to process and generate natural language. However, as they are trained on massive datasets of text, LLMs can inherit harmful biases and produce outputs that are not aligned with human values. This paper studies two main approaches to LLM alignment: Reinforcement Learning with Human Feedback (RLHF) and contrastive learning-based methods like Direct Preference Optimization (DPO). By analyzing the stability and robustness of RLHF and DPO, we propose MPO (Mixed Preference Optimization), a novel method that mitigates the weaknesses of both approaches. Specifically, we propose a two-stage training procedure: first train DPO on an easy dataset, and then perform RLHF on a difficult set with DPO model being the reference model. Here, the easy and difficult sets are constructed by a well-trained reward model that splits response pairs into those with large gaps of reward (easy), and those with small gaps (difficult). The first stage allows us to obtain a relatively optimal policy (LLM) model quickly, whereas the second stage refines LLM with online RLHF, thus mitigating the distribution shift issue associated with DPO. Experiments are conducted on two public alignment datasets, namely HH-RLHF and TLDR, demonstrating the effectiveness of MPO, both in terms of GPT4 and human evaluation.
Length-Controlled Margin-Based Preference Optimization without Reference Model
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization
OpenGVLab(开放智能机器研究实验室);Shanghai AI Laboratory(上海人工智能实验室);Fudan University(复旦大学);Nanjing University(南京大学);The Chinese University of Hong Kong(香港中文大学);Tsinghua University(清华大学);SenseTime Research(商汤科技研究院)
OpenChat: Advancing Open-source Language Models with Mixed-Quality Data
Tsinghua University(清华大学);Institute for AI Industry Research (AIR), Tsinghua University(清华大学人工智能产业研究院(AIR));Shanghai Artificial Intelligence Laboratory(上海人工智能实验室)
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);Nanyang Technological University(南洋理工大学)
提出自参考 AI 反馈框架,让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据,显著提升强化学习对齐效果。
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
Graduate School of AI, KAIST(韩国科学技术院人工智能研究生院);Samsung Advanced Institute of Technology(三星先进技术研究院);Department of Electrical and Computer Engineering, UBC(不列颠哥伦比亚大学电气与计算机工程系)
CheXalign: Preference fine-tuning in chest X-ray interpretation models without human feedback
Stanford University(斯坦福大学);University of Oxford(牛津大学);University Hospital Zurich, University of Zurich(苏黎世大学医院、苏黎世大学);Yongin Severance Hospital, Yonsei University(永临松医院、延世大学)