Direct Preference Optimization (DPO) and its variants have become increasingly popular for aligning language models with human preferences. These methods aim to teach models to better distinguish between chosen (or preferred) and rejected (or dispreferred) responses. However, prior research has identified that the probability of chosen responses often decreases during training, and this phenomenon is known as likelihood displacement. To tackle this challenge, in this work we introduce DPO-Shift to controllably shift the distribution of the chosen probability. Then, we show that DPO-Shift exhibits a fundamental trade-off between improving the chosen probability and sacrificing the reward margin, as supported by both theoretical analysis and experimental validation. Furthermore, we demonstrate the superiority of DPO-Shift over DPO on downstream tasks such as MT-Bench and a designed win rate experiment. We believe this study shows that the likelihood displacement issue of DPO can be effectively mitigated with a simple, theoretically grounded solution. Our code is available at https://github.com/Meaquadddd/DPO-Shift.
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University(宾夕法尼亚州立大学人工智能研究实验室);University of Chinese Academy of Sciences(中国科学院大学);Tencent AI Lab(腾讯人工智能实验室)
D2PO: Discriminator-Guided DPO with Response Evaluation Models
The University of Texas at Austin(德克萨斯大学奥斯汀分校);Allen Institute for Artificial Intelligence(艾伦人工智能研究所);University of Massachusetts Amherst(马萨诸塞大学阿默斯特分校);Princeton University(普林斯顿大学)
Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering
University of Minnesota(明尼苏达大学);Virginia Tech(弗吉尼亚理工大学);Xi’an University of Technology(西安理工大学);Lahore University of Management Sciences(拉合尔管理科学大学);University of California, Santa Cruz(加州大学圣克鲁兹分校)
Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));The Hong Kong University of Science and Technology(香港科技大学);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学计算机科学与工程保罗·G·艾伦学校);Max Planck Institute for Intelligent Systems, Tübingen, Germany(德国图宾根马克斯·普朗克智能系统研究所);Jinesis Lab, University of Toronto & Vector Institute(多伦多大学Jinesis实验室及向量研究所)
Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Xi’an Jiaotong University(西安交通大学);The Chinese University of Hong Kong(香港中文大学);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学)
Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization
Department of Applied Mathematics and Statistics(应用数学与统计学系);Stony Brook University(石溪大学);Institute for Advanced Computational Science(先进计算科学研究所);Center of Excellence in Wireless and Information Technology (CEWIT)(无线与信息技术卓越中心);AI Innovation Institute(人工智能创新研究院);Bloomberg(彭博);Toronto, ON M5J 2S1(多伦多,ON M5J 2S1);San Francisco, CA 94105(旧金山,CA 94105);Bloomberg New York, NY 10022(纽约,NY 10022)
FocalPO: Enhancing Preference Optimizing by Focusing on Correct Preference Rankings
LMU Munich(慕尼黑大学);Columbia University(哥伦比亚大学);University of Southern California(南加州大学);University of Oxford(牛津大学);Munich Center for Machine Learning(慕尼黑机器学习中心)
针对 DPO 难以纠正错误排序偏好对的问题,本文提出 FocalPO,通过 Focal Loss 式调制因子动态降权错误样本、强化正确样本,并在多个基准和模型上优于 DPO 变体。
Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
Centre for Machine Intelligence and Data Science, IIT Bombay(印度理工学院班加罗尔机器智能与数据科学中心);Koita Centre for Digital Health, IIT Bombay(印度理工学院班加罗尔数字健康中心);Department of Computer Science and Engineering, IIT Bombay(印度理工学院班加罗尔计算机科学与工程系)