arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.01544cs.AI
收藏

通过去噪过程奖励推进扩散语言模型的推理能力

Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards

Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)

Shaoan Xie, Lingjing Kong, Xiangchen Song, Xinshuai Dong, Guangyi Chen, Eric P. Xing, Kun Zhang

AI总结:

本文提出去噪过程奖励,通过优化去噪轨迹提升扩散语言模型的推理稳定性与可解释性,实验显示在推理任务中表现更优。

摘要:

基于大型语言模型的扩散方法提供了一种非自回归的文本生成替代方案,但使它们执行复杂推理仍具挑战性。强化学习最近 emerged 作为一种有效的后训练策略来提高其性能;然而,现有方法主要依赖于结果导向的奖励,这些奖励无法直接监督去噪过程,通常导致结构不佳的推理,难以解释且不一致地支持最终预测。为解决这一限制,我们引入去噪过程奖励,一种定义在扩散语言模型去噪轨迹上的过程级强化信号。该奖励通过估计中间去噪间隔对最终任务结果的贡献,鼓励模型偏好一致引导生成朝正确预测的推理轨迹。我们进一步提出一种高效的随机估计器,利用标准训练回放,实现大规模的过程级监督。在具有挑战性的推理基准测试中,我们的方法在推理稳定性、可解释性和整体任务性能方面均取得一致改进。

英文摘要:

Diffusion-based large language models offer a non-autoregressive alternative for text generation, but enabling them to perform complex reasoning remains challenging. Reinforcement learning has recently emerged as an effective post-training strategy for improving their performance; however, existing methods rely primarily on outcome-based rewards, which provide no direct supervision over the denoising process and often result in poorly structured reasoning that is difficult to interpret and inconsistently supports the final prediction. To address this limitation, we introduce \emph{denoising process reward}, a process-level reinforcement signal defined over the denoising trajectory of diffusion language models. This reward is obtained by estimating the contribution of intermediate denoising intervals to the final task outcome, encouraging the model to favor reasoning trajectories that consistently guide generation toward correct predictions. We further propose an efficient stochastic estimator that reuses standard training rollouts, enabling practical process-level supervision at scale. Experiments on challenging reasoning benchmarks demonstrate that our approach yields consistent improvements in reasoning stability, interpretability, and overall task performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑