arXiv 2505.104462025-10-31cs.CL

使用扩散语言模型强化横向思维的扩散思维链

Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models

Zhejiang Univeristy（浙江大学）；MAPLE Lab, Westlake University（西湖大学MAPLE实验室）；Matterwave Intelligence；Institute of Advanced Technology, Westlake Institute for Advanced Study（西湖研究所以高级技术研究所）

Zemin Huang, Zhiyang Chen, Zijun Wang, Tiancheng Li, Guo-Jun Qi

备注：: Accepted to NeurIPS 2025. Code link: https://github.com/maple-research-lab/LLaDOU

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出横向思维扩散链（DCoLT）推理框架，通过强化学习优化扩散语言模型的逆向扩散中间步骤，在数学和代码生成任务上显著提升了模型推理准确率。

摘要：

我们引入了横向思维扩散链（DCoLT），一种用于扩散语言模型的推理框架。DCoLT将逆向扩散过程中的每个中间步骤视为潜在的“思考”动作，并优化整个推理轨迹，以通过基于结果的强化学习（RL）最大化最终答案正确性的奖励。不同于遵循因果、线性思维过程的传统思维链（CoT）方法，DCoLT允许双向、非线性推理，且对其思维中间步骤的语法正确性无严格规则。我们在两个代表性的扩散语言模型（DLM）上实现了DCoLT。首先，我们选择SEDD作为代表性的连续时间离散扩散模型，其具体得分推导出一个概率策略，以在中间扩散步骤的整个序列上最大化RL奖励。我们进一步考虑离散时间掩码扩散语言模型——LLaDA，并发现预测和取消掩码token的顺序在优化由Plackett-Luce模型定义的基于排序的取消掩码策略模块（UPM）产生的RL动作中起着至关重要的作用。在数学和代码生成任务上的实验表明，仅使用公开数据和16块H800 GPU，DCoLT增强的DLM优于通过SFT或RL甚至两者结合训练的其他DLM。值得注意的是，DCoLT增强的LLaDA在GSM8K、MATH、MBPP和HumanEval上的推理准确率分别提升了+9.8%、+5.7%、+11.4%和+19.5%。

英文摘要：

We introduce the Diffusion Chain of Lateral Thought (DCoLT), a reasoning framework for diffusion language models. DCoLT treats each intermediate step in the reverse diffusion process as a latent "thinking" action and optimizes the entire reasoning trajectory to maximize the reward on the correctness of the final answer with outcome-based Reinforcement Learning (RL). Unlike traditional Chain-of-Thought (CoT) methods that follow a causal, linear thinking process, DCoLT allows bidirectional, non-linear reasoning with no strict rule on grammatical correctness amid its intermediate steps of thought. We implement DCoLT on two representative Diffusion Language Models (DLMs). First, we choose SEDD as a representative continuous-time discrete diffusion model, where its concrete score derives a probabilistic policy to maximize the RL reward over the entire sequence of intermediate diffusion steps. We further consider the discrete-time masked diffusion language model -- LLaDA, and find that the order to predict and unmask tokens plays an essential role to optimize its RL action resulting from the ranking-based Unmasking Policy Module (UPM) defined by the Plackett-Luce model. Experiments on both math and code generation tasks show that using only public data and 16 H800 GPUs, DCoLT-reinforced DLMs outperform other DLMs trained by SFT or RL or even both. Notably, DCoLT-reinforced LLaDA boosts its reasoning accuracy by +9.8%, +5.7%, +11.4%, +19.5% on GSM8K, MATH, MBPP, and HumanEval.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。