arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2604.18518cs.CVcs.LG
收藏

UDM-GRPO:面向均匀离散扩散模型的稳定高效组相对策略优化

UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models

Beijing University of Posts(北京邮电大学);Beijing Academy of Artificial Intelligence(北京人工智能研究院)

Jiaqi Wang, Haoge Deng, Ting Pan, Yang Liu, Chengyuan Wang, Fan Zhang, Yonggang Qi, Xinlong Wang

AI总结:

针对均匀离散扩散模型(UDM)与强化学习(RL)集成时训练不稳定、性能提升有限的问题,提出UDM-GRPO框架,通过将最终干净样本作为动作、利用扩散前向过程重建轨迹以及引入简化步数和无CFG策略,显著提升文本到图像生成任务的性能。

摘要:

均匀离散扩散模型(UDM)最近成为离散生成建模的一种有前景的范式;然而,其与强化学习的集成仍然很大程度上未被探索。我们观察到,将GRPO直接应用于UDM会导致训练不稳定和边际性能提升。为了解决这个问题,我们提出了UDM-GRPO,这是第一个将UDM与RL集成的框架。我们的方法基于两个关键见解:(i)将最终干净样本作为动作提供更准确和稳定的优化信号;(ii)通过扩散前向过程重建轨迹更好地将概率路径与预训练分布对齐。此外,我们引入了两种策略,即简化步数(Reduced-Step)和无CFG(CFG-Free),以进一步提高训练效率。UDM-GRPO在多个T2I任务上显著提升了基础模型性能。值得注意的是,GenEval准确率从69%提高到96%,PickScore从20.46增加到23.81,在连续和离散设置中均达到了最先进的性能。在OCR基准测试中,准确率从8%提高到57%,进一步验证了我们方法的泛化能力。代码可在https://github.com/Yovecent/UDM-GRPO获取。

英文摘要:

Uniform Discrete Diffusion Model (UDM) has recently emerged as a promising paradigm for discrete generative modeling; however, its integration with reinforcement learning remains largely unexplored. We observe that naively applying GRPO to UDM leads to training instability and marginal performance gains. To address this, we propose UDM-GRPO, the first framework to integrate UDM with RL. Our method is guided by two key insights: (i) treating the final clean sample as the action provides more accurate and stable optimization signals; and (ii) reconstructing trajectories via the diffusion forward process better aligns probability paths with the pretraining distribution. Additionally, we introduce two strategies, Reduced-Step and CFG-Free, to further improve training efficiency. UDM-GRPO significantly improves base model performance across multiple T2I tasks. Notably, GenEval accuracy improves from $69\%$ to $96\%$ and PickScore increases from $20.46$ to $23.81$, achieving state-of-the-art performance in both continuous and discrete settings. On the OCR benchmark, accuracy rises from $8\%$ to $57\%$, further validating the generalization ability of our method. Code is available at https://github.com/Yovecent/UDM-GRPO.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑