Multi-negative preference optimization under the Plackett--Luce (PL) model extends Direct Preference Optimization (DPO) by leveraging comparative signals across one preferred and multiple rejected responses. However, optimizing over large negative pools is costly, and many candidates contribute redundant gradients due to their similar effects on policy updates. We introduce MASS-DPO, a multi-negative active sample selection method that derives a PL-specific Fisher-information objective for selecting compact, informative negative subsets within each prompt. The resulting log-determinant objective selects negatives that contribute complementary information for policy updates, yielding compact subsets that retain the full pool's information while reducing redundancy. In practice, this favors negatives whose gradients cover different update directions, reducing redundant signal from near-duplicate candidates while preserving the most useful training information. Across four benchmarks spanning recommendation and multiple-choice QA and three model families, MASS-DPO consistently exceeds or matches existing methods in accuracy, improves Recall/NDCG and margin-based optimization dynamics, and delivers stronger alignment with substantially fewer negatives.
Mitigating Mismatch within Reference-based Preference Optimization
Sydney AI Centre, The University of Sydney(悉尼人工智能中心,悉尼大学);CFAR, A*STAR(CFAR,A*STAR);CSIRO, Data61(CSIRO,Data61);Southeast University(东南大学);Nanyang Technological University(南洋理工大学)
DDO-RM: Distribution-Level Policy Improvement after Reward Learning
Department of Computer Science(计算机科学系);Columbia University(哥伦比亚大学);Department of Management Science and Engineering(管理科学与工程系);Tsinghua University(清华大学);Carnegie Mellon University(卡内基梅隆大学)
P$^2$-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization
Guangdong Provincial Key Laboratory of Computational AI Models and Cognitive Intelligence, School of Computer Science & Engineering, South China University of Technology(广东省计算人工智能模型与认知智能重点实验室,计算机科学与工程学院,华南理工大学);Pazhou Lab, Guangzhou, China(琶洲实验室,广州,中国);Engineering Research Center of the Ministry of Education on Health Intelligent Perception and Paralleled Digital-Human, Guangzhou, China(教育部健康智能感知与并行数字人工程研究中心,广州,中国)
Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization
Department of Applied Mathematics and Statistics(应用数学与统计学系);Stony Brook University(石溪大学);Institute for Advanced Computational Science(先进计算科学研究所);Center of Excellence in Wireless and Information Technology (CEWIT)(无线与信息技术卓越中心);AI Innovation Institute(人工智能创新研究院);Bloomberg(彭博);Toronto, ON M5J 2S1(多伦多,ON M5J 2S1);San Francisco, CA 94105(旧金山,CA 94105);Bloomberg New York, NY 10022(纽约,NY 10022)
University of Sydney(悉尼大学);Imperial College London(伦敦帝国理工学院);University of North Carolina at Chapel Hill(北卡罗来纳大学教堂山分校);Virginia Tech(弗吉尼亚理工大学);University of Texas at Dallas(德克萨斯大学达拉斯分校)
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University(宾夕法尼亚州立大学人工智能研究实验室);University of Chinese Academy of Sciences(中国科学院大学);Tencent AI Lab(腾讯人工智能实验室)
AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education(大数据与人工智能交通运输联合实验室,(北京交通大学)教育部);School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China(计算机科学与技术学院,北京交通大学,北京,中国);Tencent Inc, China(腾讯公司,中国)
Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering
University of Minnesota(明尼苏达大学);Virginia Tech(弗吉尼亚理工大学);Xi’an University of Technology(西安理工大学);Lahore University of Management Sciences(拉合尔管理科学大学);University of California, Santa Cruz(加州大学圣克鲁兹分校)
Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gains
EPFL(苏黎世联邦理工学院);Amazon AGI(亚马逊人工智能实验室);University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Cornell University(康奈尔大学);Qualcomm AI Research(高通人工智能研究)