Direct Preference Optimization (DPO) controls the trade-off between fitting preference labels and staying close to a reference model using a single global temperature beta, implicitly treating all preference pairs as equally informative. Real-world preference corpora are heterogeneous: they mix high-signal, objective failures (for example, safety, factuality, instruction violations) with low-signal or subjective distinctions (for example, style), and also include label noise. We introduce our method, SP2DPO (Semantic Per-Pair DPO), a generalization that replaces the global temperature with an instance-specific schedule beta_i pre-decided offline from structured semantic-gap annotations (category, magnitude, confidence) produced by teacher language models. We instantiate this procedure on the UltraFeedback preference corpus (59,960 pairs), enabling large-scale construction of an auditable beta_i artifact, and incur zero training-time overhead: the inner-loop optimizer remains standard DPO with beta set per pair. We focus our empirical study on AlpacaEval 2.0, reporting both raw win rate and length-controlled win rate. Across four open-weight, instruction-tuned student backbones (4B-8B), SP2DPO is competitive with a tuned global-beta DPO baseline and improves AlpacaEval 2.0 length-controlled win rate on two of four backbones, while avoiding per-model beta sweeps. All code, annotations, and artifacts will be released.
Reinforcing Step-level Reasoning for Effective Self-Correction in LLMs
Nanyang Technological University(南洋理工大学);National University of Singapore(新加坡国立大学);VinUniversity;Institute for Infocomm Research (IR), A*STAR(新加坡科技研究局信息通信研究院)
Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Xi’an Jiaotong University(西安交通大学);The Chinese University of Hong Kong(香港中文大学);University of Chinese Academy of Sciences(中国科学院大学);Tsinghua University(清华大学);Huazhong University of Science and Technology(华中科技大学)
Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap
Paul G. Allen School of Computer Science & Engineering, University of Washington(华盛顿大学计算机科学与工程保罗·G·艾伦学校);Max Planck Institute for Intelligent Systems, Tübingen, Germany(德国图宾根马克斯·普朗克智能系统研究所);Jinesis Lab, University of Toronto & Vector Institute(多伦多大学Jinesis实验室及向量研究所)
University of Sydney(悉尼大学);Imperial College London(伦敦帝国理工学院);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Virginia Tech(弗吉尼亚理工大学);University of Texas at Dallas(德克萨斯大学达拉斯分校)
Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm
Centre for Machine Intelligence and Data Science, IIT Bombay(印度理工学院班加罗尔机器智能与数据科学中心);Koita Centre for Digital Health, IIT Bombay(印度理工学院班加罗尔数字健康中心);Department of Computer Science and Engineering, IIT Bombay(印度理工学院班加罗尔计算机科学与工程系)
P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology(广东省计算人工智能模型与认知智能重点实验室,计算机科学与工程学院,华南理工大学);Pazhou Lab, Guangzhou, China(琶洲实验室,广州,中国);Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China(教育部健康智能感知与并行数字人工程研究中心,广州,中国)
daDPO: Distribution-Aware DPO for Distilling Conversational Abilities
State Key Laboratory for Novel Software Technology, Nanjing University(南京大学新型软件技术国家重点实验室);School of Artificial Intelligence, Nanjing University(南京大学人工智能学院);University of Chicago(芝加哥大学);The Chinese University of Hong Kong(香港中文大学);East China Normal University(华东师范大学)
Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Wenge Technology(文生科技);Fudan University(复旦大学)