Mitigating Mismatch within Reference-based Preference Optimization
Sydney AI Centre, The University of Sydney（悉尼人工智能中心，悉尼大学）；CFAR, A*STAR（CFAR，A*STAR）；CSIRO, Data61（CSIRO，Data61）；Southeast University（东南大学）；Nanyang Technological University（南洋理工大学）
Suqin Yuan, Xingrui Yu, Jiyang Zheng, Lei Feng, Dadong Wang, Ivor Tsang, Tongliang Liu
Direct Preference Optimization (DPO) has become the de facto standard for offline preference alignment of large language models, but its reliance on a reference policy introduces a critical tension. DPO weighs each update relative to a reference, which stabilizes the training by regularizing the updates within a trusted region. This reliance becomes problematic for pessimistic pairs, where the reference model prefers the rejected response. For these pairs, DPO prematurely attenuates the gradient as soon as the policy margin ($Δ_θ$) merely beats the reference margin ($Δ_{\mathrm{ref}}$) even if the policy is still wrong ($Δ_θ<0$). We name this failure premature satisfaction, which is a concrete form of the training-inference mismatch. Reference-free objectives remove this mismatch by optimizing the absolute margin, but at the cost of discarding the stabilizing signal of the reference. We mitigate this tension with Hybrid-DPO (HyPO), a drop-in modification to DPO that applies reference conditionally: HyPO behaves exactly like DPO when the reference is optimistic or neutral, and it treats the reference as neutral when it is pessimistic by replacing $Δ_θ-Δ_{\mathrm{ref}}$ with $Δ_θ-\max\{0,Δ_{\mathrm{ref}}\}$. This one-line change strictly strengthens per-example learning signals on pessimistic pairs while preserving DPO's objective form and computational cost. By conditionally debiasing the pessimistic reference signal, HyPO mitigates premature satisfaction; empirically, across preference alignment, HyPO improves inference-aligned metrics and achieves higher pairwise win rates. Our results provide evidence that direct preference alignment could be enhanced by conditionally debiasing the reference signal, rather than discarding it.
Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization
Shandong University（山东大学）；Southern University of Science and Technology（南方科技大学）；University of Georgia（佐治亚大学）；National University of Singapore（新加坡国立大学）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））
Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)（人工智能与数字经济广东实验室）；Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）；Baidu Inc.（百度公司）
Quantifying and Mitigating Self-Preference Bias of LLM Judges
CompleX Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, China（复杂实验室、计算机科学与工程学院、电子科技大学、成都、中国）
Mitigating Visual Hallucinations via Semantic Curriculum Preference Optimization in MLLMs
School of Engineering, Westlake University, Hangzhou, China（西湖大学工程学院）；School of Cyberspace Security, Nanjing University of Science and Technology, Nanjing, China（南京理工大学网络安全学院）
TCPO: Thought-Centric Preference Optimization for Effective Embodied Decision-making
Tsinghua University（清华大学）；Meituan（美团）；Northeastern University（东北大学）；Beijing University of Posts and Telecommunications（北京邮电大学）；Meta AI；Wuhan University（武汉大学）
ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）；University of Illinois Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）；Korea University（韩国大学）
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought
Tianjin Key Laboratory of Cognitive Computing（天津认知计算实验室）；Tianjin University（天津大学）；Huiyan Technology Company, Ltd.（慧颜科技有限公司）；Chinese Academy of Sciences（中国科学院）；Tencent（腾讯）
Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning
Hong Kong University of Science and Technology（香港科技大学）；University of Southern California（南加州大学）；Tencent AI Lab（腾讯AI实验室）；University of California, Davis（加州大学戴维斯分校）
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；Department of Computer Science and Technology, Tsinghua University（清华大学计算机科学与技术系）；Tencent Inc.（腾讯公司）
Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing
City University of Hong Kong（香港城市大学）；City University of Hong Kong Shenzhen Research Institute（香港城市大学深圳研究院）；University of Science and Technology of China（中国科学技术大学）；Zhejiang University（浙江大学）
DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity
The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））；New Jersey Institute of Technology（新泽西理工学院）；Institute of Computing Technology, CAS（中国科学院计算技术研究所）
Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering
University of Minnesota（明尼苏达大学）；Virginia Tech（弗吉尼亚理工大学）；Xi’an University of Technology（西安理工大学）；Lahore University of Management Sciences（拉合尔管理科学大学）；University of California, Santa Cruz（加州大学圣克鲁兹分校）
