Direct Preference Optimization (DPO) has emerged as a simple and effective method for aligning large language models. However, its reliance on a fixed temperature parameter leads to suboptimal training on diverse preference data, causing overfitting on easy examples and under-learning from informative ones. Recent methods have emerged to counter this. While IPO addresses general overfitting, its uniform regularization can be overly conservative. The more targeted approach of $β$-DPO suffers from its own limitations: its batch-level adaptation applies a single, compromised temperature to mixed-margin pairs, its linear update rule can produce unstable negative $β$ values, and its filtering mechanism discards potentially useful training signals. In this work, we introduce Margin-Adaptive Direct Preference Optimization (MADPO), a method that provides a stable, data-preserving, and instance-level solution. MADPO employs a practical two-step approach: it first trains a reward model to estimate preference margins and then uses these margins to apply a continuous, adaptive weight to the DPO loss for each individual training sample. This re-weighting scheme creates an effective target margin that is amplified for hard pairs and dampened for easy pairs, allowing for granular control over the learning signal. We provide a comprehensive theoretical analysis, proving that MADPO has a well-behaved optimization landscape and is robust to reward model estimation errors. We validate our theory with experiments on a summarization task using human preference data. MADPO consistently outperforms strong baselines across a comprehensive sweep of decoding temperatures.
Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)(人工智能与数字经济广东实验室);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学人工智能学院);Baidu Inc.(百度公司)
Mitigating Mismatch within Reference-based Preference Optimization
Sydney AI Centre, The University of Sydney(悉尼人工智能中心,悉尼大学);CFAR, A*STAR(CFAR,A*STAR);CSIRO, Data61(CSIRO,Data61);Southeast University(东南大学);Nanyang Technological University(南洋理工大学)
HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
Vellore Institute of Technology(维洛雷理工学院);University of Massachusetts Amherst(马萨诸塞大学阿姆赫斯特分校);Northwestern University(西北大学);Yale University(耶鲁大学);Algoverse AI Research(Algoverse AI研究)
Less is More: Improving LLM Alignment via Preference Data Selection
University of Science and Technology of China(中国科学技术大学);Peking University(北京大学);Massachusetts Institute of Technology(麻省理工学院);Alibaba Cloud Computing(阿里云计算);MoE Key Lab of BIPC, University of Science and Technology of China(中国科学技术大学MoE关键实验室)
Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
Centre for Machine Intelligence and Data Science, IIT Bombay(印度理工学院班加罗尔机器智能与数据科学中心);Koita Centre for Digital Health, IIT Bombay(印度理工学院班加罗尔数字健康中心);Department of Computer Science and Engineering, IIT Bombay(印度理工学院班加罗尔计算机科学与工程系)
RePO: Understanding Preference Learning Through ReLU-Based Optimization
University of Science and Technology of China(中国科学技术大学);Alibaba Group(阿里巴巴集团);Institute of Dataspace, Hefei Comprehensive National Science Center(数据空间研究所,合肥综合性国家科学中心);MoE Key Lab of BIPC, University of Science and Technology of China(BIPC联合实验室,中国科学技术大学)
When Do Model Internals Help? Exploring the Role of Representation Engineering in LLM Safety
State Key Laboratory of Multimedia Information Processing, Peking University(北京大学多媒体信息处理国家重点实验室);School of Computer Science, Peking University(北京大学计算机科学学院)
EditCaption: Human-Refined SFT and HAE-DPO for Image Editing Instruction Synthesis
Peking University(北京大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Tsinghua University(清华大学);Beihang University(北京航空航天大学);Xiaohongshu Inc.(小红书公司)
Weights-Rotated Preference Optimization for Large Language Models
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络安全学院);Baidu Inc.(百度公司)
Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
Shandong University(山东大学);Southern University of Science and Technology(南方科技大学);University of Georgia(佐治亚大学);National University of Singapore(新加坡国立大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
School of Computer Science, Peking University(北京大学计算机科学系);Center on Frontiers of Computing Studies, Peking University(北京大学前沿计算研究中心);National Research and Engineering Center of Software Engineering, Peking University(北京大学软件工程研究中心)
DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)
Google Research(谷歌研究);Google DeepMind(谷歌DeepMind);Work done while at Google DeepMind(在谷歌深Mind的工作);CEE-M, Univ Montpellier, CNRS, INRAE, Institut Agro(CEE-M,蒙彼利埃大学,CNRS,INRAE,农业研究所)
Listwise Direct Preference Optimization with Multi-Dimensional Preference Mixing
University of Alberta(阿尔伯塔大学);University of Toronto(多伦多大学);Zhejiang University(浙江大学);School of Computer Science McGill University(麦吉尔大学计算机学院);Alibaba Group (Ant Group)(阿里巴巴集团(蚂蚁集团));Chinese Academy of Sciences(中国科学院)
DDO-RM: Distribution-Level Policy Improvement after Reward Learning
Department of Computer Science(计算机科学系);Columbia University(哥伦比亚大学);Department of Management Science and Engineering(管理科学与工程系);Tsinghua University(清华大学);Carnegie Mellon University(卡内基梅隆大学)
Harness Local Rewards for Global Benefits: Effective Text-to-Video Generation Alignment with Patch-level Reward Models
School of Computer Science, Carnegie Mellon University(卡内基梅隆大学计算机科学学院);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Serendipity One Inc.(Serendipity One公司)