AlphaGRPO：通过分解性可验证奖励解锁UMMs中的自反思多模态生成
AlphaGRPO: Unlocking Self-Reflective Multimodal Generation in UMMs via Decompositional Verifiable Reward
AI总结：
AlphaGRPO通过分解性可验证奖励提升多模态生成能力，实现文本到图像生成和自反思修正，验证了自反思强化方法在生成高质量输出中的有效性。
摘要：
在本文中，我们提出AlphaGRPO，一种将组相对策略优化（GRPO）应用于AR-Diffusion统一多模态模型（UMMs）的新型框架，以增强多模态生成能力，无需额外的冷启动阶段。我们的方法解锁了模型内在潜力，使其能够执行高级推理任务：推理文本到图像生成，其中模型主动推断隐含用户意图，以及自反思修正，其中模型自主诊断并纠正生成输出中的不一致。为解决提供稳定监督以实现实世界多模态生成的挑战，我们引入分解性可验证奖励（DVReward）。不同于整体标量奖励，DVReward利用LLM将复杂用户请求分解为原子、可验证的语义和质量问题，然后由通用MLLM评估，提供可靠且可解释的反馈。广泛的实验表明，AlphaGRPO在多模态生成基准测试中，包括GenEval、TIIF-Bench、DPG-Bench和WISE，均取得了稳健的改进，同时在不训练编辑任务的情况下，在GEdit任务上也取得了显著进展。这些结果验证了我们的自反思强化方法有效利用内在理解来指导高质量生成。项目页面：https://huangrh99.github.io/AlphaGRPO/
英文摘要：
In this paper, we propose AlphaGRPO, a novel framework that applies Group Relative Policy Optimization (GRPO) to AR-Diffusion Unified Multimodal Models (UMMs) to enhance multimodal generation capabilities without an additional cold-start stage. Our approach unlocks the model's intrinsic potential to perform advanced reasoning tasks: Reasoning Text-to-Image Generation, where the model actively infers implicit user intents, and Self-Reflective Refinement, where it autonomously diagnoses and corrects misalignments in generated outputs. To address the challenge of providing stable supervision for real-world multimodal generation, we introduce the Decompositional Verifiable Reward (DVReward). Unlike holistic scalar rewards, DVReward utilizes an LLM to decompose complex user requests into atomic, verifiable semantic and quality questions, which are then evaluated by a general MLLM to provide reliable and interpretable feedback. Extensive experiments demonstrate that AlphaGRPO yields robust improvements across multimodal generation benchmarks, including GenEval, TIIF-Bench, DPG-Bench and WISE, while also achieving significant gains in editing tasks on GEdit without training on editing tasks. These results validate that our self-reflective reinforcement approach effectively leverages inherent understanding to guide high-fidelity generation. Project page: https://huangrh99.github.io/AlphaGRPO/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
