Direct Preference Optimization (DPO) aligns language models using pairwise preference comparisons, offering a simple and effective alternative to Reinforcement Learning (RL) from human feedback. However, in many practical settings, training data consists of multiple rollouts per prompt, inducing rich preference structure that pairwise DPO fails to exploit. Collapsing such data into independent pairs discards transitivity, introduces redundant or conflicting supervision, and can lead to unstable optimization. We propose Graph Direct Preference Optimization (GraphDPO), a structured generalization of DPO that operates over directed acyclic preference graphs induced by rollout rankings. GraphDPO encodes dominance relations as edges and optimizes a graph-structured Plackett--Luce-inspired objective that aggregates supervision over dominated neighborhoods, propagating the transitive structure of the input preferences while recovering standard DPO as a special case. GraphDPO naturally handles tied and incomparable responses through an equivalence-class construction: responses with identical preference signals form graph layers, incomparable responses share a layer without an edge, and absent edges contribute zero loss, preventing the spurious gradients that strict total-order objectives incur. Under this layered construction, GraphDPO maintains linear per-prompt complexity via efficient log-sum-exp aggregation. We further incorporate optional ground-truth anchoring by inserting verified solutions as dominant nodes and applying an annealed schedule that stabilizes early training while gradually relaxing oracle supervision. Experiments on reasoning and program synthesis tasks demonstrate superior performance, suggesting that graph-structured preference modeling is a scalable and robust alternative to pairwise and listwise alignment objectives.
In the Markov decision process model, policies are usually evaluated by expected cumulative rewards. As this decision criterion is not always suitable, we propose in this paper an algorithm for computing a policy optimal for the quantile criterion. Both finite and infinite horizons are considered. Finally we experimentally evaluate our approach on random MDPs and on a data center control problem.
Radiology Report Generation via Multi-objective Preference Optimization
East China University of Science and Technology（华东理工大学）；Harbin Institute of Technology（哈尔滨工业大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
National Taiwan University（国立台湾大学）；Texas A&M University（德克萨斯A&M大学）；Appier AI Research（Appier人工智能研究院）；Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）
DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)
Google Research（谷歌研究）；Google DeepMind（谷歌DeepMind）；Work done while at Google DeepMind（在谷歌深Mind的工作）；CEE-M, Univ Montpellier, CNRS, INRAE, Institut Agro（CEE-M，蒙彼利埃大学，CNRS，INRAE，农业研究所）
Multi-Agent Reinforcement Learning via Agent-Specific Preference
Tsinghua University（清华大学）；Chinese Academy of Sciences（中国科学院）；CFINS；BNRist；Institute for Embodied Intelligence and Robotics（嵌入式智能与机器人研究所）；Institute of Automation（自动化研究所）
Center for Intelligent and Networked System (CFINS)（智能与网络化系统中心）；Department of Automation（自动化系）；Beijing National Research Center for Information Science and Technology（北京信息科学与技术国家研究中心）；Beijing Key Laboratory of Embodied Intelligence Systems（北京具身智能系统重点实验室）；Tsinghua University（清华大学）
Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator
Department of Statistics and Data Science, Southern University of Science and Technology（统计与数据科学系，南方科技大学）；Department of Computer Sciences, University of Wisconsin-Madison（计算机科学系，威斯康星大学麦迪逊分校）
Many real-world applications are characterized by a number of conflicting performance measures. As optimizing in a multi-objective setting leads to a set of non-dominated solutions, a preference function is required for selecting the solution with the appropriate trade-off between the objectives. The question is: how good do estimations of these objectives have to be in order for the solution maximizing the preference function to remain unchanged? In this paper, we introduce the concept of preference radius to characterize the robustness of the preference function and provide guidelines for co…
Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution
School of Computer Science, Fudan University（复旦大学计算机科学学院）；College of Foreign Languages and Literature, Fudan University（复旦大学外国语言文学学院）；Institute of Modern Languages and Linguistics, Fudan University（复旦大学现代语言与语言学研究所）
HCRide: Harmonizing Passenger Fairness and Driver Preference for Human-Centered Ride-Hailing
Department of Computer Science, Florida State University（佛罗里达州立大学计算机科学系）；Department of Computer Science and Engineering, Lehigh University（莱斯利大学计算机科学与工程系）
Many circumstances of practical importance have performance or success metrics which exist implicitly---in the eye of the beholder, so to speak. Tuning aspects of such problems requires working without defined metrics and only considering pairwise comparisons or rankings. In this paper, we review an existing Bayesian optimization strategy for determining most-preferred outcomes, and identify an adaptation to allow it to handle ties. We then discuss some of the issues we have encountered when humans use this optimization strategy to optimize coloring a piece of abstract artwork. We hope that, b…
