从视频与模拟中学习以动作和推理为中心的图像编辑
Learning Action and Reasoning-Centric Image Editing from Videos and Simulations
AI总结：
本文针对现有图像编辑模型在动作和推理类编辑上的不足，构建了来自视频与模拟的高质量 AURORA 数据集和 AURORA-Bench 基准，并提出新的判别性自动评估指标，微调后的模型在 8 类编辑任务上显著优于以往方法。
摘要：
一个图像编辑模型应当能够执行多样化的编辑，从物体替换、改变属性或风格，到执行动作或运动，这些都需要多种形式的推理。当前通用的指令引导编辑模型在以动作和推理为中心的编辑上存在显著不足。物体、属性或风格的变化可以从视觉静态数据集中学习。另一方面，用于以动作和推理为中心的编辑的高质量数据十分稀缺，并且必须来自完全不同的来源，这些来源涵盖例如物理动力学、时间性和空间推理。为此，我们精心构建了 AURORA 数据集（Action-Reasoning-Object-Attribute），这是一个高质量训练数据集合，由人工标注并从视频和模拟引擎中整理而来。我们关注高质量训练数据的一个关键方面：三元组（源图像、提示、目标图像）包含由提示描述的单一有意义的视觉变化，即源图像和目标图像之间真正最小的变化。为了证明我们数据集的价值，我们在一个新的专家整理的基准（AURORA-Bench）上评估了经过 AURORA 微调的模型，该基准涵盖 8 个不同的编辑任务。根据人类评分者的判断，我们的模型显著优于以往的编辑模型。对于自动评估，我们发现以往指标存在重要缺陷，并提醒在语义困难的编辑任务中谨慎使用它们。相反，我们提出了一种新的自动指标，专注于判别性理解。我们希望我们的努力：（1）整理高质量训练数据集和评估基准，（2）开发关键评估方法，以及（3）发布最先进的模型，将推动通用图像编辑的进一步发展。
英文摘要：
An image editing model should be able to perform diverse edits, ranging from object replacement, changing attributes or style, to performing actions or movement, which require many forms of reasoning. Current general instruction-guided editing models have significant shortcomings with action and reasoning-centric edits. Object, attribute or stylistic changes can be learned from visually static datasets. On the other hand, high-quality data for action and reasoning-centric edits is scarce and has to come from entirely different sources that cover e.g. physical dynamics, temporality and spatial reasoning. To this end, we meticulously curate the AURORA Dataset (Action-Reasoning-Object-Attribute), a collection of high-quality training data, human-annotated and curated from videos and simulation engines. We focus on a key aspect of quality training data: triplets (source image, prompt, target image) contain a single meaningful visual change described by the prompt, i.e., truly minimal changes between source and target images. To demonstrate the value of our dataset, we evaluate an AURORA-finetuned model on a new expert-curated benchmark (AURORA-Bench) covering 8 diverse editing tasks. Our model significantly outperforms previous editing models as judged by human raters. For automatic evaluations, we find important flaws in previous metrics and caution their use for semantically hard editing tasks. Instead, we propose a new automatic metric that focuses on discriminative understanding. We hope that our efforts : (1) curating a quality training dataset and an evaluation benchmark, (2) developing critical evaluations, and (3) releasing a state-of-the-art model, will fuel further progress on general image editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
