arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.00931cs.LGcs.AI
收藏

连续效用直接偏好优化

Continuous-Utility Direct Preference Optimization

stanford(斯坦福大学);meta;glasgow(格拉斯哥大学)

Muhammad Ahmed Mohsin, Muhammad Umer, Emily Fox

AI总结:

本文提出CU-DPO框架,通过连续评分替代二元标签,提升模型在数学推理任务中的策略选择准确率和推理能力。

摘要:

大型语言模型的推理常被视为单一能力,依赖二元偏好监督无法捕捉部分进展或细粒度推理质量。我们引入连续效用直接偏好优化(CU-DPO)框架,通过替换二元标签为连续评分,对齐模型到一组基于提示的认知策略。我们证明学习K种策略可使样本复杂度提升Theta(K log K)倍,并证明DPO收敛到熵正则化的效用最大化策略。为利用此信号,我们提出两阶段训练流程:(i) 策略选择,通过最佳对比较优化模型选择给定问题的最佳策略;(ii) 执行细化,通过边际分层对训练模型正确执行选定策略。在数学推理基准上,CU-DPO将策略选择准确率从35-46%提升到68-78%,在分布内数据集上获得一致的推理增益,有效迁移至分布外任务。

英文摘要:

Large language model reasoning is often treated as a monolithic capability, relying on binary preference supervision that fails to capture partial progress or fine-grained reasoning quality. We introduce continuous utility direct preference optimization (CU-DPO), a framework that aligns models to a portfolio of prompt-based cognitive strategies by replacing binary labels with continuous scores that capture fine-grained reasoning quality. We prove that learning with K strategies yields a Theta(K log K) improvement in sample complexity over binary preferences and that DPO converges to the entropy-regularized utility-maximizing policy. To exploit this signal, we propose a two-stage pipeline: (i) strategy selection, which optimizes the model to choose the best strategy via best-vs-all comparisons, and (ii) execution refinement, which trains correct execution using margin-stratified pairs. The framework is domain-agnostic: any task admitting cognitively distinct solution strategies and a decomposable continuous utility signal can be incorporated into the portfolio. On mathematical reasoning benchmarks, CU-DPO improves strategy selection accuracy from 35-46% to 68-78% across seven base models, yielding downstream reasoning gains of up to +6.6 points on in-distribution datasets with effective out-of-distribution transfer. CU-DPO demonstrates consistent gains on code generation and causal reasoning benchmarks, confirming generalization beyond the mathematical domain.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑