Large Language Models (LLMs) have demonstrated remarkable potential in automating software development tasks. While recent advances leverage Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) to align models with human preferences, the optimal training strategy remains unclear across diverse code preference scenarios. This paper systematically investigates the roles of SFT and DPO in aligning LLMs with different code preferences. Through both theoretical analysis and empirical observation, we hypothesize that SFT excels in scenarios with objectively verifiable optimal solutions, while applying SFT followed by DPO (S&D) enables models to explore superior solutions in scenarios without objectively verifiable optimal solutions. Based on the analysis and experimental evidence, we propose Adaptive Preference Optimization (APO), a dynamic integration approach that adaptively amplifies preferred responses, suppresses dispreferred ones, and encourages exploration of potentially superior solutions during training. Extensive experiments across six representative code preference tasks validate our theoretical hypotheses and demonstrate that APO consistently matches or surpasses the performance of existing SFT and S&D strategies. Our work provides both theoretical foundations and practical guidance for selecting appropriate training strategies in different code preference alignment scenarios.
Preference Orchestrator: Prompt-Aware Multi-Objective Alignment for Large Language Models
School of Computer Science and Engineering(计算机科学与工程学院);Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications(新一代人工智能技术及其交叉应用关键实验室);Ministry of Education(教育部)
Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models
The Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Soochow University(苏州大学);University of Technology Sydney(悉尼科技大学);The Hong Kong University of Science and Technology(香港科技大学)
针对成对偏好蒸馏忽视回复差异程度的问题,本文提出 PAD 框架,将师生内在偏好建模为分布并进行对齐,在多个基准上显著提升 SLM 的人类偏好对齐效果。
Alignment Imprint: Zero-Shot AI-Generated Text Detection via Provable Preference Discrepancy
Nankai University(南开大学);Tsinghua University(清华大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Laboratory(鹏城实验室);Wuhan University(武汉大学)
Differentially Private Preference Data Synthesis for Large Language Model Alignment
Department of Computer Science, University of Virginia, Charlottesville, Virginia, USA(弗吉尼亚大学计算机科学系);Department of Electrical and Computer Engineering, University of Virginia, Charlottesville, Virginia, USA(弗吉尼亚大学电气与计算机工程系)
Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence, Beijing University of Posts and Telecommunications(北京邮电大学人工智能学院)
AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selection
Southern University of Science and Technology(南方科技大学);Peking University(北京大学);MSRA(微软亚洲研究院);Shanghai University of Finance and Economics(上海财经大学);Tsinghua University(清华大学)
FlipGuard: Defending Preference Alignment against Update Regression with Constrained Optimization
University of Science and Technology of China(中国科学技术大学);State Key Laboratory of Communication Content Cognition(通信内容认知国家重点实验室);Beijing University of Posts and Telecommunications(北京邮电大学)
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系);Tencent Inc.(腾讯公司)
Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
Graduate School of AI, POSTECH, Pohang, Republic of Korea(POSTECH人工智能研究生院);Department of CSE, POSTECH, Pohang, Republic of Korea(POSTECH计算机科学与工程系);National AI Research Lab, Seoul, Republic of Korea(首尔国家人工智能研究实验室)
Instant Preference Alignment for Text-to-Image Diffusion Models
New Laboratory of Pattern Recognition, CASIA(模式识别新实验室,中国科学院自动化研究所);The Hong Kong University of Science and Technology(香港科技大学);Nanjing university(南京大学);Academy of Broadcasting Science, NRTA(广播科学研究院,国家广播电视总局)
BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization
Graduate School of AI, KAIST(韩国科学技术院人工智能研究生院);Samsung Advanced Institute of Technology(三星先进技术研究院);Department of Electrical and Computer Engineering, UBC(不列颠哥伦比亚大学电气与计算机工程系)