arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2601.22385cs.CLcs.AIcs.LG
收藏

SP^2DPO: 一种基于大语言模型的语义逐对DPO泛化

SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization

Alibaba--NTU Global e-Sustainability CorpLab (ANGEL), Singapore(阿里-NTU全球可持续性公司实验室(ANGEL),新加坡);Alibaba Group, China(阿里集团,中国)

Chaoyue He, Xin Zhou, Di Wang, Hong Xu, Wei Liu, Chunyan Miao

AI总结:

SP^2DPO通过语义逐对DPO泛化提升AlpacaEval 2.0的长度控制胜率,避免每模型beta扫描。

摘要:

直接偏好优化(DPO)通过一个全局温度beta控制拟合偏好标签和保持参考模型之间的平衡,隐式地将所有偏好对视为等信息量。现实中的偏好语料库是异质的:它们混合了高信号、客观的失败(例如,安全、事实性、指令违反)与低信号或主观的区别(例如,风格),并包含标签噪声。我们提出了一种方法,即SP2DPO(语义逐对DPO),这是一种泛化,将全局温度替换为实例特定的计划beta_i,该beta_i在结构化语义间隙注释(类别、幅度、置信度)上预先决定,由教师语言模型生成。我们在此过程中实例化UltraFeedback偏好语料库(59,960对),使能够大规模构建可审计的beta_i artifact,并且训练时间开销为零:内循环优化器仍保持标准DPO,beta设置为每对。我们专注于AlpacaEval 2.0的实证研究,报告了原始胜率和长度控制胜率。在四个开放权重、指令微调的学生主干(4B-8B)上,SP2DPO与调优的全局beta DPO基线具有竞争力,并在两个主干上提高了AlpacaEval 2.0长度控制胜率,同时避免了每个模型beta扫描。所有代码、注释和artifacts都将发布。

英文摘要:

Direct Preference Optimization (DPO) controls the trade-off between fitting preference labels and staying close to a reference model using a single global temperature beta, implicitly treating all preference pairs as equally informative. Real-world preference corpora are heterogeneous: they mix high-signal, objective failures (for example, safety, factuality, instruction violations) with low-signal or subjective distinctions (for example, style), and also include label noise. We introduce our method, SP2DPO (Semantic Per-Pair DPO), a generalization that replaces the global temperature with an instance-specific schedule beta_i pre-decided offline from structured semantic-gap annotations (category, magnitude, confidence) produced by teacher language models. We instantiate this procedure on the UltraFeedback preference corpus (59,960 pairs), enabling large-scale construction of an auditable beta_i artifact, and incur zero training-time overhead: the inner-loop optimizer remains standard DPO with beta set per pair. We focus our empirical study on AlpacaEval 2.0, reporting both raw win rate and length-controlled win rate. Across four open-weight, instruction-tuned student backbones (4B-8B), SP2DPO is competitive with a tuned global-beta DPO baseline and improves AlpacaEval 2.0 length-controlled win rate on two of four backbones, while avoiding per-model beta sweeps. All code, annotations, and artifacts will be released.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑