Direct Preference Optimization (DPO) has become a widely used training method for the instruction fine-tuning of large language models (LLMs). In this work, we explore an under-investigated aspect of DPO - its dependency on the reference model or policy. Such reference policies, typically instantiated as the model to be further fine-tuned, are important since they can impose an upper limit on DPO's effectiveness. Therefore, we address three related research questions in this work. First, we explore the optimal strength of the KL divergence constraint in DPO, which penalizes deviations from the reference policy, and find that DPO is sensitive to this strength. Next, we examine the necessity of the KL-constraint from the reference policies in DPO by providing both theoretical and empirical comparisons between DPO and related learning objectives, demonstrating DPO's superiority in this controlled setting. Additionally, we investigate whether DPO benefits from stronger reference policies, finding that a stronger reference policy can lead to improved performance, but only when it is similar to the model being fine-tuned. Our findings highlight the confounding role of reference policies in DPO and offer insights for best practices, while also identifying open research questions for future studies.
MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
Department of Statistics, Rutgers University, New Brunswick, United States;Department of Computer Science, Rutgers University, New Brunswick, United States;College of Management of Technology, EPFL, Switzerland;Department of Computer Science, ETH Zurich, Switzerland
Length-Controlled Margin-Based Preference Optimization without Reference Model
School of Artificial Intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China(知识驱动人机智能工程研究中心,教育部,中国);International Center of Future Science, Jilin University(未来科学国际中心,吉林大学)
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
Graduate School of AI, KAIST(韩国科学技术院人工智能研究生院);Samsung Advanced Institute of Technology(三星先进技术研究院);Department of Electrical and Computer Engineering, UBC(不列颠哥伦比亚大学电气与计算机工程系)
Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization
Department of Applied Mathematics and Statistics(应用数学与统计学系);Stony Brook University(石溪大学);Institute for Advanced Computational Science(先进计算科学研究所);Center of Excellence in Wireless and Information Technology (CEWIT)(无线与信息技术卓越中心);AI Innovation Institute(人工智能创新研究院);Bloomberg(彭博);Toronto, ON M5J 2S1(多伦多,ON M5J 2S1);San Francisco, CA 94105(旧金山,CA 94105);Bloomberg New York, NY 10022(纽约,NY 10022)
Aligning Large Language Models from Self-Reference AI Feedback with one General Principle
School of Computer Science, Fudan University(复旦大学计算机科学学院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);The University of Sydney(悉尼大学);Nanyang Technological University(南洋理工大学)
提出自参考 AI 反馈框架,让 Llama2-Chat 依据通用原则、自我批评与自一致性生成高质量偏好数据,显著提升强化学习对齐效果。