ADIEE：用于指令引导图像编辑评估的自动数据集创建与评分器
ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluation
AI总结：
针对指令引导图像编辑缺乏公开训练数据与可靠自动评估的问题，提出 ADIEE 自动构建超10万样本数据集并训练数值评分器，显著提升相关性、成对比较准确率及编辑模型表现。
摘要：
指令引导图像编辑的最新进展凸显了有效自动化评估的必要性。尽管视觉-语言模型（VLM）已被探索用作评判器，但开源模型在对齐方面存在困难，而专有模型缺乏透明度和成本效率。此外，目前不存在用于微调开源 VLM 的公开训练数据集，只有采用多样化评估方案的小型基准。为解决这一问题，我们提出 ADIEE，这是一种自动化数据集创建方法，随后用于训练指令引导图像编辑评估的评分模型。我们生成了一个包含超过 10 万个样本的大规模数据集，并用它微调 LLaVA-NeXT-8B 模型；该模型经过修改，可从自定义 token 解码出数值评分。所得评分器在所有基准上均优于所有开源 VLM 和 Gemini-Pro 1.5：在 AURORA-Bench 上，其评分与人类评级的相关性提升 0.0696（+17.24%）；与最先进方法相比，在 GenAI-Bench 和 AURORA-Bench 上的成对比较准确率分别提高 4.03%（+7.21%）和 4.75%（+9.35%）。该评分器可充当奖励模型，实现自动化最佳编辑选择和模型微调。值得注意的是，所提出的评分器可将 MagicBrush 模型在 ImagenHub 上的平均评估得分从 5.90 提升至 6.43（+8.98%）。我们的代码和模型可在 https://github.com/SherryXTChen/ADIEE.git 获取。
英文摘要：
Recent advances in instruction-guided image editing underscore the need for effective automated evaluation. While Vision-Language Models (VLMs) have been explored as judges, open-source models struggle with alignment, and proprietary models lack transparency and cost efficiency. Additionally, no public training datasets exist to fine-tune open-source VLMs, only small benchmarks with diverse evaluation schemes. To address this, we introduce ADIEE, an automated dataset creation approach which is then used to train a scoring model for instruction-guided image editing evaluation. We generate a large-scale dataset with over 100K samples and use it to fine-tune a LLaVA-NeXT-8B model modified to decode a numeric score from a custom token. The resulting scorer outperforms all open-source VLMs and Gemini-Pro 1.5 across all benchmarks, achieving a 0.0696 (+17.24%) gain in score correlation with human ratings on AURORA-Bench, and improving pair-wise comparison accuracy by 4.03% (+7.21%) on GenAI-Bench and 4.75% (+9.35%) on AURORA-Bench, respectively, compared to the state-of-the-art. The scorer can act as a reward model, enabling automated best edit selection and model fine-tuning. Notably, the proposed scorer can boost MagicBrush model's average evaluation score on ImagenHub from 5.90 to 6.43 (+8.98%). Our code and models are available at https://github.com/SherryXTChen/ADIEE.git.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
