基于离散扩散模型的视觉-文本思维高效强化学习
Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Model
AI总结：
提出用离散扩散模型替代自回归模型进行多模态强化学习，通过局部视觉编辑减少计算量，并设计分解奖励分配策略解决跨模态干扰问题。
摘要：
基于强化学习的后训练已被广泛采用，以在能够同时进行文本和图像生成的统一多模态模型中实现交错视觉和文本推理。然而，大多数现有方法建立在自回归统一模型上，在视觉推理过程中需要完整的图像再生。在这项工作中，我们证明多模态离散扩散模型是自回归模型在交错推理中进行强化学习的有效替代方案，因为它们能够通过局部视觉编辑而非完整的图像令牌再生来执行高效的视觉展开。与自回归基线相比，这使GRPO期间的展开计算减少了26.9%，且性能下降极小。尽管效率提高，我们发现联合奖励分配（在模态间使用共享奖励信号）在RL更新期间会在不相关的图像和文本令牌序列之间引入跨模态干扰。为解决此问题，我们提出分解奖励分配策略，该策略独立地为文本和视觉片段分配奖励。采用分解奖励分配后，我们的RL方法相比联合奖励分配提高了11.2%，相比基础模型提高了38.04%。
英文摘要：
RL-based post-training has been widely adopted to enable interleaved visual and textual reasoning in unified multimodal models capable of both text and image generation. However, most existing approaches are built upon autoregressive (AR) unified models, which require full image regeneration during visual reasoning. In this work, we demonstrate that multimodal discrete diffusion models are effective alternatives to AR models for reinforcement learning in interleaved reasoning, owing to their ability to perform efficient visual rollouts via localized visual editing rather than full image-token regeneration. This reduces rollout computation during GRPO by 26.9\% compared to AR baselines, with minimal performance drop. Despite the improved efficiency, we find that joint reward assignment, which employs a shared reward signal across modalities, introduces cross-modal interference between unrelated image and text token sequences during RL updates. To address this issue, we propose factorized reward assignment, a strategy that assigns rewards independently to text and vision segments. With factorized reward assignment, our RL approach achieves an 11.2% improvement over joint reward assignment and a 38.04% improvement over the base model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
