利用基于验证器的强化学习进行图像编辑
Leveraging Verifier-Based Reinforcement Learning in Image Editing
AI总结：
本文提出Edit-R1框架，通过构建基于推理的验证器奖励模型（RRM）来解决图像编辑中缺乏稳健奖励模型的问题，该模型通过分解指令为不同原则并逐项评估图像，实现细粒度奖励，实验表明其在图像编辑任务中优于现有模型。
摘要：
尽管强化学习从人类反馈（RLHF）已成为文本到图像生成的关键范式，但其在图像编辑中的应用仍鲜有研究。关键瓶颈在于缺乏适用于所有编辑任务的稳健通用奖励模型。现有编辑奖励模型通常仅提供总体评分而无详细检查，忽视了不同指令要求，导致奖励偏差。为此，我们主张从简单的评分器转向推理验证器。我们引入Edit-R1框架，构建基于推理链（CoT）的验证器奖励模型（RRM）并用于下游图像编辑。Edit-RRM将指令分解为不同的原则，将编辑后的图像与每个原则进行评估，并将这些检查汇总成可解释、细粒度的奖励。为了构建此类RRM，我们首先应用监督微调（SFT）作为“冷启动”生成CoT奖励轨迹。然后，我们引入组对比偏好优化（GCPO），一种利用人类配对偏好数据强化点状RRM的强化学习算法。在构建RRM后，我们使用GRPO训练编辑模型，利用此非可微但强大的奖励模型。大量实验表明，我们的Edit-RRM在图像编辑特定任务中优于强大的VLMs如Seed-1.5-VL和Seed-1.6-VL，并观察到明显的扩展趋势，性能从3B到7B参数持续提升。此外，Edit-R1为编辑模型如FLUX.1-kontext带来增益，凸显了其在提升图像编辑任务中的有效性。
英文摘要：
While Reinforcement Learning from Human Feedback (RLHF) has become a pivotal paradigm for text-to-image generation, its application to image editing remains largely unexplored. A key bottleneck is the lack of a robust general reward model for all editing tasks. Existing edit reward models usually give overall scores without detailed checks, ignoring different instruction requirements and causing biased rewards. To address this, we argue that the key is to move from a simple scorer to a reasoning verifier. We introduce Edit-R1, a framework that builds a chain-of-thought (CoT) verifier-based reasoning reward model (RRM) and then leverages it for downstream image editing. The Edit-RRM breaks instructions into distinct principles, evaluates the edited image against each principle, and aggregates these checks into an interpretable, fine-grained reward. To build such an RRM, we first apply supervised fine-tuning (SFT) as a ``cold-start'' to generate CoT reward trajectories. Then, we introduce Group Contrastive Preference Optimization (GCPO), a reinforcement learning algorithm that leverages human pairwise preference data to reinforce our pointwise RRM. After building the RRM, we use GRPO to train editing models with this non-differentiable yet powerful reward model. Extensive experiments demonstrate that our Edit-RRM surpasses powerful VLMs such as Seed-1.5-VL and Seed-1.6-VL as an editing-specific reward model, and we observe a clear scaling trend, with performance consistently improving from 3B to 7B parameters. Moreover, Edit-R1 delivers gains to editing models like FLUX.1-kontext, highlighting its effectiveness in enhancing image editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
