arXiv 2601.151652026-06-08cs.CLcs.AIcs.LG

灵活性陷阱：重新思考扩散语言模型中任意顺序的价值

The Flexibility Trap: Rethinking the Value of Arbitrary Order in Diffusion Language Models

LeapLab, Tsinghua University（清华大学Leap实验室）；NLPLab, Tsinghua University（清华大学自然语言处理实验室）；Tsinghua University（清华大学）；Alibaba Group（阿里巴巴集团）；BNRist, Tsinghua University（清华大学北京研究院）

Zanlin Ni, Shenzhi Wang, Yang Yue, Tianyu Yu, Weilin Zhao, Yeguo Hua, Tianyi Chen, Jun Song, Cheng Yu, Bo Zheng, Gao Huang

备注：: Code and pre-trained models: https://github.com/LeapLabTHU/JustGRPO

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文发现，尽管扩散语言模型（dLLMs）允许任意生成顺序，但这种灵活性可能限制其推理能力，通过采用标准的Group Relative Policy Optimization（GRPO）方法，即JustGRPO，在保持并行解码能力的同时提升了推理性能。

摘要：

扩散大语言模型（dLLMs）打破了传统语言模型的严格左到右约束，使token生成可以按任意顺序进行。直观上，这种灵活性意味着解决方案空间严格超越了固定的自回归轨迹，理论上解锁了更强大的推理潜力。然而，在本文中，我们发现对于一般推理任务（例如数学和编程），任意顺序生成可能实际上会限制dLLMs的推理潜力。我们观察到dLLMs倾向于利用这种顺序灵活性来绕过关键探索的高不确定性token，这可能导致解决方案覆盖的过早崩溃。这一观察促使我们重新思考dLLMs的强化学习方法，其中大量的复杂性，如处理组合轨迹和不可计算的似然，通常致力于保持这种灵活性。我们证明，通过放弃任意顺序并应用标准的Group Relative Policy Optimization（GRPO）方法，即JustGRPO，可以有效地激发推理能力。我们的方法，JustGRPO，虽然简洁却出人意料地有效（例如在GSM8K上达到89.1%的准确率），同时完全保留了dLLMs的并行解码能力。项目页面：https://nzl-thu.github.io/the-flexibility-trap

英文摘要：

Diffusion Large Language Models (dLLMs) break the rigid left-to-right constraint of traditional LLMs, enabling token generation in arbitrary orders. Intuitively, this flexibility implies a solution space that strictly supersets the fixed autoregressive trajectory, theoretically unlocking superior reasoning potential. However, in this paper, we find that for general reasoning tasks (e.g., mathematics and coding), arbitrary order generation may in fact limit the reasoning potential of dLLMs. We observe that dLLMs tend to exploit this order flexibility to bypass high-uncertainty tokens that are crucial for exploration, which can lead to a premature collapse of solution coverage. This observation motivates a rethink of RL approaches for dLLMs, where considerable complexities, such as handling combinatorial trajectories and intractable likelihoods, are often devoted to preserving this flexibility. We show that effective reasoning can be elicited by simply forgoing arbitrary order and applying standard Group Relative Policy Optimization (GRPO) instead. Our approach, JustGRPO, is minimalist yet surprisingly effective (e.g., 89.1% accuracy on GSM8K) while fully retaining the parallel decoding ability of dLLMs. Project page: https://nzl-thu.github.io/the-flexibility-trap

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。