arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.09291cs.LGstat.AP
收藏

dFlowGRPO:面向离散流模型的速率感知策略优化

dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models

Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);East China Normal University(东华大学);University of California, Los Angeles(加州大学洛杉矶分校);University of Toronto(多伦多大学)

Zhengyan Wan, Yidong Ouyang, Panwen Hu, Qiang Sun

AI总结:

本文提出dFlowGRPO框架,用于离散流模型的强化学习,支持多种概率路径和非掩码源分布,通过轨迹概率推导和去噪马尔可夫决策过程,提升文本到图像生成和多模态理解任务性能。

摘要:

离散流模型(DFMs)是一类灵活的生成模型,用于生成离散数据,扩散大语言模型(dLLMs)可视为其特例。尽管已有研究将强化学习应用于dLLMs,但将其扩展到更一般的离散流模型仍显不足。本文提出离散流-GRPO(dFlowGRPO),一种统一的强化学习框架,支持广泛的概率路径和非掩码源分布。我们推导了DFMs的完整轨迹概率,并将去噪建模为马尔可夫决策过程,使dFlowGRPO能够结合关联的条件转移率和后验模型信息。我们将其应用于FUDOKI,一个近期的多模态离散流模型,并在图像生成和多模态理解任务上进行评估。实验结果表明,dFlowGRPO在文本到图像生成任务中优于现有GRPO型方法,并在与使用FlowGRPO训练的连续流模型相竞争的同时,展示了在理解任务上的强大能力。

英文摘要:

Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with a specific choice of a mixture path and a masked source distribution. While several recent works have explored reinforcement learning for dLLMs, its application to more general discrete flow models remains underexplored. In this work, we present discrete Flow-GRPO (dFlowGRPO), a unified reinforcement learning framework for discrete flow models that supports a broad family of probability paths and non-masked source distributions. We derive the full trajectory probability for DFMs and formulate the denoising process as a Markov decision process, enabling dFlowGRPO to incorporate information from both the associated conditional transition rates and the posterior model during reinforcement learning. We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks. Empirical results show that dFlowGRPO outperforms existing GRPO-type methods for dLLMs on text-to-image generation tasks and achieves performance competitive with continuous flow-based models trained using Flow-GRPO, while also demonstrating strong capabilities on understanding tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑