arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2507.20109cs.SEcs.AI
收藏

学习对齐人类代码偏好

Learning to Align Human Code Preferences

Zhejiang University(浙江大学)

Xin Yin, Chao Ni, Xiaohu Yang

AI总结:

本文提出自适应偏好优化(APO)方法,通过动态整合SFT和DPO,提升模型在不同代码偏好场景下的对齐性能。

摘要:

大型语言模型(LLMs)在自动化软件开发任务中展现了显著的潜力。尽管最近的进展利用监督微调(SFT)和直接偏好优化(DPO)来对齐模型与人类偏好,但在多样化的代码偏好场景中,最佳的训练策略仍然不明确。本文系统地研究了SFT和DPO在对齐LLMs与不同代码偏好中的作用。通过理论分析和实证观察,我们假设SFT在具有客观可验证最优解的场景中表现优异,而应用SFT后接DPO(S&D)使模型能够探索更优解的场景中没有客观可验证的最优解。基于分析和实验证据,我们提出了自适应偏好优化(APO),一种动态整合方法,该方法能够自适应地放大受青睐的响应,抑制不受欢迎的响应,并在训练过程中鼓励探索潜在更优的解。在六个代表性的代码偏好任务上进行的广泛实验验证了我们的理论假设,并证明APO在一致性和表现上与现有的SFT和S&D策略相匹配或超越。我们的工作为在不同代码偏好对齐场景中选择适当训练策略提供了理论基础和实践指导。

英文摘要:

Large Language Models (LLMs) have demonstrated remarkable potential in automating software development tasks. While recent advances leverage Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) to align models with human preferences, the optimal training strategy remains unclear across diverse code preference scenarios. This paper systematically investigates the roles of SFT and DPO in aligning LLMs with different code preferences. Through both theoretical analysis and empirical observation, we hypothesize that SFT excels in scenarios with objectively verifiable optimal solutions, while applying SFT followed by DPO (S&D) enables models to explore superior solutions in scenarios without objectively verifiable optimal solutions. Based on the analysis and experimental evidence, we propose Adaptive Preference Optimization (APO), a dynamic integration approach that adaptively amplifies preferred responses, suppresses dispreferred ones, and encourages exploration of potentially superior solutions during training. Extensive experiments across six representative code preference tasks validate our theoretical hypotheses and demonstrate that APO consistently matches or surpasses the performance of existing SFT and S&D strategies. Our work provides both theoretical foundations and practical guidance for selecting appropriate training strategies in different code preference alignment scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑