EditVal：对基于扩散的文本引导图像编辑方法进行基准测试
EditVal: Benchmarking Diffusion Based Text-Guided Image Editing Methods
AI总结：
该研究提出标准化基准EditVal，含筛选数据集、13类编辑属性与自动化评估流水线，对8种前沿扩散文本引导图像编辑方法测试并结合人类研究验证相关性，得出性能差异与现存问题，公开发布相关资源以推动领域发展。
摘要：
近期，研究人员利用Imagen、Stable Diffusion等大规模扩散生成模型的强大能力，开发出了大量文本引导图像编辑方法。然而目前尚无标准化评估协议，可对不同类型细粒度编辑的方法进行横向比较。为填补这一空白，我们提出EditVal——一个用于定量评估文本引导图像编辑方法的标准化基准。EditVal包含经过精心筛选的图像数据集、为每张图像设置的源自13种编辑类型的可编辑属性，以及一套自动化评估流水线，该流水线借助预训练视觉-语言模型，评估生成图像在每种编辑类型下的保真度。我们使用EditVal对8种前沿扩散编辑方法（包括SINE、Imagic和Instruct-Pix2Pix）开展基准测试。我们还开展了大规模人类研究，结果表明EditVal的自动化评估流水线与我们所研究编辑类型的人类偏好高度相关。结合人类研究与自动化评估结果，我们发现：（1）在不同编辑类型的平均表现上，Instruct-Pix2Pix、Null-Text和SINE是性能最优的方法，但仅Instruct-Pix2Pix和Null-Text能够保留原始图像的属性；（2）大多数编辑方法在涉及空间操作的编辑任务（例如改变物体位置）中表现不佳；（3）不存在能在多种不同编辑类型中均排名第一的“全能最优”方法。我们希望该基准能为未来开发更可靠的文本引导图像编辑工具奠定基础。我们将公开发布EditVal及所有相关代码、人类研究模板，相关资源可访问https://deep-ml-research.github.io/editval/获取，以支持相关研究方向的发展。
英文摘要：
A plethora of text-guided image editing methods have recently been developed by leveraging the impressive capabilities of large-scale diffusion-based generative models such as Imagen and Stable Diffusion. A standardized evaluation protocol, however, does not exist to compare methods across different types of fine-grained edits. To address this gap, we introduce EditVal, a standardized benchmark for quantitatively evaluating text-guided image editing methods. EditVal consists of a curated dataset of images, a set of editable attributes for each image drawn from 13 possible edit types, and an automated evaluation pipeline that uses pre-trained vision-language models to assess the fidelity of generated images for each edit type. We use EditVal to benchmark 8 cutting-edge diffusion-based editing methods including SINE, Imagic and Instruct-Pix2Pix. We complement this with a large-scale human study where we show that EditVall's automated evaluation pipeline is strongly correlated with human-preferences for the edit types we considered. From both the human study and automated evaluation, we find that: (i) Instruct-Pix2Pix, Null-Text and SINE are the top-performing methods averaged across different edit types, however {\it only} Instruct-Pix2Pix and Null-Text are able to preserve original image properties; (ii) Most of the editing methods fail at edits involving spatial operations (e.g., changing the position of an object). (iii) There is no `winner' method which ranks the best individually across a range of different edit types. We hope that our benchmark can pave the way to developing more reliable text-guided image editing tools in the future. We will publicly release EditVal, and all associated code and human-study templates to support these research directions in https://deep-ml-research.github.io/editval/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
