Preference optimization methods such as DPO align large language models (LLMs) using paired comparisons, but their effectiveness can be highly sensitive to the quality and difficulty of preference pairs. A common heuristic treats small-margin (ambiguous) pairs as noisy and filters them out. In this paper, we revisit this assumption and show that pair difficulty interacts strongly with the optimization objective: when trained with preference-based losses, difficult pairs can destabilize training and harm alignment, yet these same pairs still contain useful supervision signals when optimized with supervised fine-tuning (SFT). Motivated by this observation, we propose MixDPO, a simple yet effective difficulty-aware training strategy that (i) orders preference data from easy to hard (a curriculum over margin-defined difficulty), and (ii) routes difficult pairs to an SFT objective while applying a preference loss to easy pairs. This hybrid design provides a practical mechanism to leverage ambiguous pairs without incurring the optimization failures often associated with preference losses on low-margin data. Across three LLM-judge benchmarks, MixDPO consistently improves alignment over DPO and a range of widely-used variants, with particularly strong gains on AlpacaEval~2 length-controlled (LC) win rate.
Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
Shandong University(山东大学);Southern University of Science and Technology(南方科技大学);University of Georgia(佐治亚大学);National University of Singapore(新加坡国立大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
APLOT: Robust Reward Modeling via Adaptive Preference Learning with Optimal Transport
Shenzhen International Center for Industrial and Applied Mathematics(深圳工业与应用数学国际中心);Shenzhen Research Institute of Big Data(深圳大数据研究 institute);The Chinese University of Hong Kong Shenzhen(香港中文大学(深圳));Birmingham City University(伯明翰城市大学);Jilin University(吉林大学);KAUST(科威特大学);Hefei University of Technology(合肥工业大学)
Mitigating Mismatch within Reference-based Preference Optimization
Sydney AI Centre, The University of Sydney(悉尼人工智能中心,悉尼大学);CFAR, A*STAR(CFAR,A*STAR);CSIRO, Data61(CSIRO,Data61);Southeast University(东南大学);Nanyang Technological University(南洋理工大学)
RePO: Understanding Preference Learning Through ReLU-Based Optimization
University of Science and Technology of China(中国科学技术大学);Alibaba Group(阿里巴巴集团);Institute of Dataspace, Hefei Comprehensive National Science Center(数据空间研究所,合肥综合性国家科学中心);MoE Key Lab of BIPC, University of Science and Technology of China(BIPC联合实验室,中国科学技术大学)
Less is More: Improving LLM Alignment via Preference Data Selection
University of Science and Technology of China(中国科学技术大学);Peking University(北京大学);Massachusetts Institute of Technology(麻省理工学院);Alibaba Cloud Computing(阿里云计算);MoE Key Lab of BIPC, University of Science and Technology of China(中国科学技术大学MoE关键实验室)
TPO: Aligning Large Language Models with Multi-branch & Multi-step Preference Trees
School of Computer Science, Peking University(北京大学计算机科学系);Center on Frontiers of Computing Studies, Peking University(北京大学前沿计算研究中心);National Research and Engineering Center of Software Engineering, Peking University(北京大学软件工程研究中心)
An incremental preference elicitation-based approach to learning potentially non-monotonic preferences in multi-criteria sorting
Dalian University of Technology(大连理工大学);University of Alberta(阿尔伯塔大学);Systems Research Institute, Polish Academy of Sciences(波兰科学院系统研究所);Istinye University(伊斯提涅大学)
Act-Adaptive Margin: Dynamically Calibrating Reward Models for Subjective Ambiguity
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences(深圳先进技术研究院,中国科学院);University of New South Wales(新南威尔士大学);Tongyi Lab, Alibaba Group(通义实验室,阿里巴巴集团)
SimulPL: Aligning Human Preferences in Simultaneous Machine Translation
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Graduate School of Translation and Interpretation, Beijing Foreign Studies University(北京外国语大学高级翻译学院)
Property-driven Protein Inverse Folding With Multi-Objective Preference Alignment
School of Computer Science, Peking University(北京大学计算机科学系);BioGeometry;Mila - Québec AI Institute(魁北克人工智能研究所);Université de Montréal(蒙特利尔大学);HEC Montréal(蒙特利尔HEC商学院);CIFAR AI Research Chair(CIFAR人工智能研究主席)
Beyond Uniform Forgetting: A Study of Sequential Direct Preference Optimization Across Preference Settings
Network Analysis and Social Influence Modelling (NASIM) Lab(网络分析与社会影响建模实验室);School of Physics Maths and Computing, The University of Western Australia(西澳大学物理数学与计算学院);School of Psychological Science, The University of Western Australia(西澳大学心理科学学院);School of Computing, Macquarie University(麦考瑞大学计算机学院)