arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2510.08233cs.LG
收藏

通过分布匹配策略优化增强扩散大语言模型的推理能力

Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization

Georgia Institute of Technology(佐治亚理工学院)

Yuchen Zhu, Wei Guo, Jaemoo Choi, Petr Molodyk, Bo Yuan, Molei Tao, Yongxin Chen

AI总结:

本文提出DMPO方法,通过分布匹配优化提升扩散大语言模型的推理能力,实现显著的准确率提升。

摘要:

扩散大语言模型(dLLMs)是自回归大语言模型(AR-LLMs)的有前途的替代方案,因为它们可能允许更高的推理吞吐量。强化学习(RL)是dLLMs实现与AR-LLMs在重要任务(如推理)上相媲美的关键组成部分。然而,适合dLLMs独特特性的RL算法尚未开发。本文提出了分布匹配策略优化(DMPO),一种原理上严谨且理论上有根据的RL微调方法,专门设计用于通过交叉熵优化将dLLM的策略分布匹配到最优的、奖励倾斜的分布,从而增强dLLMs的推理能力。我们识别出在小训练批大小实施中的关键挑战,并通过一种新颖的权重基线减法技术提出几种有效的解决方案。DMPO在多个推理基准测试中表现出色,无需监督微调,其准确率提升高达54.3%超过先前的SOTA基线,且达到基模型的66.41%,凸显了分布匹配框架的有效性。我们的代码可在https://github.com/yuchen-zhu-zyc/DMPO获取。

英文摘要:

Diffusion large language models (dLLMs) are promising alternatives to autoregressive large language models (AR-LLMs), as they potentially allow higher inference throughput. Reinforcement learning (RL) is crucial to enabling dLLMs to achieve performance comparable to that of AR-LLMs on important tasks, such as reasoning. However, RL algorithms well-suited to dLLMs' unique characteristics have yet to be developed. This paper proposes Distribution Matching Policy Optimization (DMPO), a principled and theoretically grounded RL fine-tuning method specifically designed to enhance the reasoning capabilities of dLLMs by matching the dLLM policy distribution to the optimal, reward-tilted one through cross-entropy optimization. We identify a key implementation challenge with small training batch sizes and propose several effective solutions based on a novel weight baseline subtraction technique. DMPO exhibits superior performance on multiple reasoning benchmarks without supervised fine-tuning, achieving up to a $39.63$ percentage-point improvement in accuracy over prior non-DMPO RL baselines and $67.97$ percentage points over the base model, underscoring the effectiveness of the distribution-matching framework. Our code is available at https://github.com/yuchen-zhu-zyc/DMPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑