EditReward：一种用于指令引导图像编辑的人类对齐奖励模型
EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing
AI总结：
EditReward是一种基于人类偏好的奖励模型，通过大规模标注数据提升图像编辑任务的质量与性能，展示了在指令引导图像编辑中的卓越对齐能力。
摘要：
最近，我们见证了自然语言指令在图像编辑领域取得巨大进展。几种闭源模型如GPT-Image-1、Seedream和Google-Nano-Banana已显示出高度有前景的进步。然而，开源模型仍然落后。主要瓶颈是缺乏可靠的奖励模型来扩展高质量合成训练数据。为了解决这个关键瓶颈，我们构建了EditReward，它使用我们新的大规模人类偏好数据集进行训练，该数据集由受过训练的专家根据包含超过20万对偏好数据的严格协议进行精心标注。EditReward在指令引导图像编辑任务中表现出对人类偏好的卓越对齐性。实验表明，EditReward在已建立的基准如GenAI-Bench、AURORA-Bench、ImagenHub和我们新的EditReward-Bench上实现了最先进的与人类的相关性，优于广泛使用的VLM-as-judge模型。此外，我们使用EditReward从现有的嘈杂ShareGPT-4o-Image数据集中选择高质量子集。我们训练Step1X-Edit在所选子集上，其在训练完整集上的表现有显著提升。这证明了EditReward作为奖励模型扩展图像编辑高质量训练数据的能力。此外，其强大的对齐性表明了在强化学习基础上进行后训练和测试时间扩展的潜在应用。EditReward及其训练数据集将被发布，以帮助社区构建更多高质量的图像编辑训练数据集。
英文摘要：
Recently, we have witnessed great progress in image editing with natural language instructions. Several closed-source models like GPT-Image-1, Seedream, and Google-Nano-Banana have shown highly promising progress. However, the open-source models are still lagging. The main bottleneck is the lack of a reliable reward model to scale up high-quality synthetic training data. To address this critical bottleneck, we built EditReward, trained with our new large-scale human preference dataset, meticulously annotated by trained experts following a rigorous protocol containing over 200K preference pairs. EditReward demonstrates superior alignment with human preferences in instruction-guided image editing tasks. Experiments show that EditReward achieves state-of-the-art human correlation on established benchmarks such as GenAI-Bench, AURORA-Bench, ImagenHub, and our new EditReward-Bench, outperforming a wide range of VLM-as-judge models. Furthermore, we use EditReward to select a high-quality subset from the existing noisy ShareGPT-4o-Image dataset. We train Step1X-Edit on the selected subset, which shows significant improvement over training on the full set. This demonstrates EditReward's ability to serve as a reward model to scale up high-quality training data for image editing. Furthermore, its strong alignment suggests potential for advanced applications like reinforcement learning-based post-training and test-time scaling of image editing models. EditReward with its training dataset will be released to help the community build more high-quality image editing training datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
