DACA-GRPO：去噪感知的信用分配用于扩散语言模型中的强化学习
DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Models
AI总结：
本文提出DACA-GRPO，通过引入去噪进度评分和分层掩码似然，改进扩散语言模型中强化学习的信用分配，提升数学推理、代码生成等任务性能。
摘要：
扩散大语言模型是自回归模型的有力替代品，但现有强化学习方法将所有去噪步骤视为同等重要，并依赖于有偏、高方差的似然估计。我们识别出两个根本性弱点：去噪轨迹中缺乏时间信用分配，以及用于策略优化的均场似然估计存在系统偏差。为了解决这些问题，我们提出了Denoising-Aware Credit Assignment for GRPO（DACA-GRPO），一种轻量级、即插即用的增强方法，适用于任何GRPO风格的训练器。DACA-GRPO引入了两个互补机制：去噪进度评分，从中间预测中提取每token的重要性权重，无需额外前向成本；分层掩码似然，将token位置分为层次，使每个token在大部分序列作为上下文的情况下进行预测，从而减少均场偏差。在三种GRPO基础方法上应用DACA-GRPO，使其在七个基准测试中取得一致提升，涵盖数学推理、代码生成、约束满足和受约束生成等任务，在数学推理中提升达5.6个百分点，在代码生成中提升7.4个百分点，在约束满足中提升36.3个百分点，在JSON schema符合性中提升5.9个百分点。
英文摘要：
Diffusion large language models are a compelling alternative to autoregressive models, yet existing RL methods for diffusion treat all denoising steps as equally important and rely on biased, high-variance likelihood estimates. We identify two fundamental weaknesses: the absence of temporal credit assignment across the denoising trajectory, and the systematic bias of mean-field likelihood estimates used for policy optimization. To address these, we propose Denoising-Aware Credit Assignment for GRPO (DACA-GRPO), a lightweight, plug-and-play enhancement for any GRPO-style trainer. DACA-GRPO introduces two complementary mechanisms: Denoising Progress Scores, which extract per-token importance weights from intermediate predictions at no additional forward cost, and Stratified Masking Likelihood, which partitions token positions into strata so that each token is predicted with most of the sequence as context, reducing the mean-field bias. Applied on top of three GRPO base methods, DACA-GRPO achieves consistent improvements across seven benchmarks spanning mathematical reasoning, code generation, constraint satisfaction, and constrained generation, with gains of up to 5.6pp on math reasoning, 7.4pp on code generation, 36.3pp on constraint satisfaction, and 5.9pp on JSON schema adherence.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
