Direct Preference Optimization (DPO) and related methods align large language models from pairwise preferences by regularizing updates against a fixed reference policy. As the policy drifts, a static reference, however, can become increasingly miscalibrated, leading to distributional mismatch and amplifying spurious preference signals under noisy supervision. Conversely, reference-free variants avoid mismatch but often suffer from unconstrained reward drift. We propose Geometric Anchor Preference Optimization (GAPO), which replaces the fixed reference with a dynamic, geometry-aware anchor: an adversarial local perturbation of the current policy within a small radius that serves as a pessimistic baseline. This anchor enables an adaptive reweighting mechanism, modulating the importance of each preference pair based on its local sensitivity. We further introduce the Anchor Gap, the reward discrepancy between the policy and its anchor, and show under smoothness conditions that it approximates worst-case local margin degradation. Optimizing a logistic objective weighted by this gap downweights geometrically brittle instances while emphasizing robust preference signals. Across diverse noise settings, GAPO consistently improves robustness while matching or improving performance on standard LLM alignment and reasoning benchmarks.
Data Selection for LLM Alignment Using Fine-Grained Preferences
National Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）；School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）；Algorithm Tech, Taobao & Tmall Group of Alibaba（阿里巴巴集团算法技术部）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
Regularized Conditional Diffusion Model for Multi-Task Preference Alignment
Harbin Institute of Technology（哈尔滨工业大学）；Institute of Artificial Intelligence (TeleAI), China Telecom（中国电信人工智能研究院（TeleAI））；Hong Kong University of Science and Technology（香港科技大学）
SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Baichuan Inc.（百川智能）
PEARL: Zero-shot Cross-task Preference Alignment and Robust Reward Learning for Robotic Manipulation
Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）；King’s College London（伦敦国王学院）；Beijing Institute for General AI（北京通用人工智能研究院）；Shanghai Jiao Tong University（上海交通大学）
Preference Alignment on Diffusion Model: A Comprehensive Survey for Image Generation and Editing
University of Liverpool（利物浦大学）；Institute of Software Chinese Academy of Sciences（中国科学院软件研究所）；University of Edinburgh（爱丁堡大学）；University of Bristol（布里斯托大学）；University of Exeter（埃克塞特大学）
Harvard University（哈佛大学）；University of California Berkeley（加州大学伯克利分校）；Massachusetts Institute of Technology（麻省理工学院）；Ikigai Labs（Ikigai实验室）；University of California San Diego（加州大学圣地亚哥分校）
Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
Artificial Intelligence Research Laboratory, Pennsylvania State University（宾夕法尼亚州立大学人工智能研究实验室）；University of Chinese Academy of Sciences（中国科学院大学）；Tencent AI Lab（腾讯人工智能实验室）
Differentially Private Preference Data Synthesis for Large Language Model Alignment
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA（弗吉尼亚大学计算机科学系）；Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA（弗吉尼亚大学电气与计算机工程系）
How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
Inria（法国国家信息与自动化研究所）；École Normale Supérieure（巴黎高等师范学校）；PSL Research University（巴黎综合理工研究所）；Northwestern University（西北大学）；University of California, Berkeley（加州大学伯克利分校）；University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）
Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors
National Taiwan University（国立台湾大学）；Texas A&M University（德克萨斯A&M大学）；Appier AI Research（Appier人工智能研究院）；Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）
ProteinOPD: Towards Effective and Efficient Preference Alignment for Protein Design
Tsinghua University（清华大学）；International Digital Economy Academy（国际数字经济学院）；Hong Kong University of Science and Technology (Guangzhou)（香港科学与技术大学（广州））；Nanyang Technological University（南洋理工大学）
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学高瓴人工智能学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）
Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
Graduate School of AI, POSTECH, Pohang, Republic of Korea（POSTECH人工智能研究生院）；Department of CSE, POSTECH, Pohang, Republic of Korea（POSTECH计算机科学与工程系）；National AI Research Lab, Seoul, Republic of Korea（首尔国家人工智能研究实验室）
