相信你的批评者：用于忠实图像编辑和生成的鲁棒奖励建模与强化学习
Trust Your Critic: Robust Reward Modeling and Reinforcement Learning for Faithful Image Editing and Generation
AI总结：
本文提出FIRM框架，通过定制数据集和鲁棒奖励模型，提升图像编辑和生成的忠实性与指令遵循，采用'基础与奖励'策略平衡一致性与质量，实现性能突破。
摘要：
强化学习（RL）已逐渐成为增强图像编辑和文本到图像（T2I）生成的有前途的范式。然而，当前的奖励模型，作为RL中的批评者，常常存在幻觉并分配出噪声分数，本质上误导了优化过程。在本文中，我们提出了FIRM（忠实图像奖励建模），一个全面的框架，旨在开发鲁棒的奖励模型，以提供准确且可靠的指导，用于忠实的图像生成和编辑。首先，我们设计了定制的数据整理管道，以构建高质量的评分数据集。具体来说，我们通过执行和一致性来评估编辑，而生成主要通过遵循指令来评估。使用这些管道，我们收集了FIRM-Edit-370K和FIRM-Gen-293K数据集，并训练了专门的奖励模型（FIRM-Edit-8B和FIRM-Gen-8B），这些模型能够准确反映这些标准。其次，我们引入了FIRM-Bench，一个专门设计用于编辑和生成批评者的综合基准。评估显示，我们的模型在与人类判断的一致性方面优于现有指标。此外，为了无缝地将这些批评者整合到RL流程中，我们提出了一种新的“基础与奖励”奖励策略，平衡了竞争目标：一致性调节执行（CME）用于编辑，质量调节一致性（QMA）用于生成。通过这一框架，我们的结果模型FIRM-Qwen-Edit和FIRM-SD3.5实现了显著的性能突破。全面的实验表明，FIRM减轻了幻觉，建立了优于现有通用模型的忠实性和指令遵循的新标准。我们所有数据集、模型和代码均已公开在https://firm-reward.github.io。
英文摘要：
Reinforcement learning (RL) has emerged as a promising paradigm for enhancing image editing and text-to-image (T2I) generation. However, current reward models, which act as critics during RL, often suffer from hallucinations and assign noisy scores, inherently misguiding the optimization process. In this paper, we present FIRM (Faithful Image Reward Modeling), a comprehensive framework that develops robust reward models to provide accurate and reliable guidance for faithful image generation and editing. First, we design tailored data curation pipelines to construct high-quality scoring datasets. Specifically, we evaluate editing using both execution and consistency, while generation is primarily assessed via instruction following. Using these pipelines, we collect the FIRM-Edit-370K and FIRM-Gen-293K datasets, and train specialized reward models (FIRM-Edit-8B and FIRM-Gen-8B) that accurately reflect these criteria. Second, we introduce FIRM-Bench, a comprehensive benchmark specifically designed for editing and generation critics. Evaluations demonstrate that our models achieve superior alignment with human judgment compared to existing metrics. Furthermore, to seamlessly integrate these critics into the RL pipeline, we formulate a novel "Base-and-Bonus" reward strategy that balances competing objectives: Consistency-Modulated Execution (CME) for editing and Quality-Modulated Alignment (QMA) for generation. Empowered by this framework, our resulting models FIRM-Qwen-Edit and FIRM-SD3.5 achieve substantial performance breakthroughs. Comprehensive experiments demonstrate that FIRM mitigates hallucinations, establishing a new standard for fidelity and instruction adherence over existing general models. All of our datasets, models, and code have been publicly available at https://firm-reward.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
