保留还是修改？文本引导图像编辑中平衡保留与修改的上下文感知评估
Preserve or Modify? Context-Aware Evaluation for Balancing Preservation and Modification in Text-Guided Image Editing
AI总结：
针对现有文本引导图像编辑评估指标存在的上下文盲、偏向单一维度的问题，提出上下文感知指标AugCLIP，通过多模态大模型增强描述、超平面计算修改向量实现自适应平衡，在5个基准上优于现有指标且更贴合人类判断。
摘要：
视觉-语言模型与生成模型的发展极大推动了文本引导图像编辑技术的进步，该技术旨在根据目标文本对图像进行修改的同时，保留源图像的核心元素。然而现有评估指标存在上下文盲的问题，对完全不同的源图像-目标文本对不加区分地采用相同评估标准，要么偏向修改程度要么偏向保留程度。方向性CLIP相似度是唯一同时考虑源图像和目标文本的指标，但它也偏向修改维度，且会关注图像中与编辑无关的区域。我们提出AugCLIP，一种上下文感知的评估指标，能够根据给定源图像和目标文本的具体上下文，自适应协调保留与修改两个维度的评估。该指标通过推导理想编辑图像的CLIP表征实现：理想编辑图像既保留源图像内容，又经过必要修改以匹配目标文本。具体而言，AugCLIP利用多模态大语言模型增强源图像和目标文本的文本描述，随后通过在CLIP空间中分隔源属性与目标属性的超平面计算修改向量。在涵盖多样编辑场景的5个基准数据集上开展的大量实验表明，AugCLIP与人类评估标准的契合度极高，性能优于现有指标。代码可在https://github.com/augclip/augclip_eval获取。
英文摘要：
The development of vision-language and generative models has significantly advanced text-guided image editing, which seeks the preservation of core elements in the source image while implementing modifications based on the target text. However, existing metrics have a context-blindness problem, indiscriminately applying the same evaluation criteria on completely different pairs of source image and target text, biasing towards either modification or preservation. Directional CLIP similarity, the only metric that considers both source image and target text, is also biased towards modification aspects and attends to irrelevant editing regions of the image. We propose AugCLIP, a context-aware metric that adaptively coordinates preservation and modification aspects, depending on the specific context of a given source image and target text. This is done by deriving the CLIP representation of an ideally edited image, that preserves the source image with necessary modifications to align with target text. More specifically, using a multi-modal large language model, AugCLIP augments the textual descriptions of the source and target, then calculates a modification vector through a hyperplane that separates source and target attributes in CLIP space. Extensive experiments on five benchmark datasets, encompassing a diverse range of editing scenarios, show that AugCLIP aligns remarkably well with human evaluation standards, outperforming existing metrics. The code is available at https://github.com/augclip/augclip_eval.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
