发表机构
MIT; ByteDance; Michigan State University（麻省理工学院; 字节跳动; 密歇根州立大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对现有模型难以理解涂鸦意图的问题，构建ScribbleEdit基准，并提出软标记基线方法，提升涂鸦图像编辑性能。
AI 中文摘要
基于涂鸦的交互方式为用户在交互式编辑工具中指定图像编辑意图提供了一种轻量且直观的方法。然而，当前基于VLM或LLM的图像编辑模型难以理解并仅根据涂鸦输入来执行编辑。为了系统地研究这一问题，我们构建了一个新的基准ScribbleEdit，用于评估图像编辑模型在涂鸦条件下执行图像编辑的能力。该任务既需要对涂鸦意图的深入理解，也需要对其空间信息的准确解读。在ScribbleEdit中，我们设计了一个自动化的数据构建流程，并引入了一个专门评估意图对齐的评估协议。我们的分析表明，现有的基于VLM/LLM的编辑模型无法准确捕捉涂鸦意图。为了指导仅基于涂鸦的图像编辑的未来进展，我们提出了一种简单而有效的软标记基线方法，该方法增强了模型对涂鸦语义的理解，并在我们的基准上优于标准的图像编辑模型。我们的评估和基线共同为评估和改进涂鸦驱动的图像编辑提供了具体的基础。
英文摘要
Scribble-based interaction provides a lightweight and intuitive way for users to specify image editing intents in interactive editing tools. However, current image editing models based on VLMs or LLMs struggle to understand and execute edits based solely on scribble inputs. To systematically study this problem, we construct a new benchmark, ScribbleEdit, that evaluates the ability of image editing models to perform image editing conditioned on scribbles. This task requires both a deep understanding of the intention of the scribble and an accurate interpretation of its spatial information. In ScribbleEdit, we design an automated data construction pipeline and introduce a dedicated evaluation protocol that explicitly measures intention alignment. Our analysis reveals that existing VLM/LLM-based editing models fail to accurately capture scribble intentions. To guide future progress on scribble-only image editing, we propose a simple yet effective soft-token baseline, which enhances the model's understanding of scribble semantics and outperforms standard image editing models on our benchmark. Our evaluation and baseline together provide a concrete foundation for assessing and improving the scribble-driven image editing.
