文本-视觉协同指导的图像编辑
Text-Vision Co-Instructed Image Editing
AI总结：
提出TV-Edit框架，联合文本指令的语义表达与稀疏视觉指令的空间引导，实现精确且忠实于意图的图像编辑，显著优于现有方法。
摘要：
现有的图像编辑方法通常可分为基于文本指令和基于视觉提示两类。文本指令语义表达丰富，但受限于编辑结果空间控制的粗粒度。相比之下，拖拽和点等视觉提示能提供精确的空间引导，但存在语义意图固有的模糊性。为统一文本和视觉提示的优势，我们提出文本-视觉协同指导的图像编辑，将文本指令作为语义意图、稀疏视觉指令作为空间引导联合建模，旨在实现精确且忠实于意图的图像操作。为此，我们首先构建了一个包含超过23K个样本的文本-视觉指令配对数据集，这些样本源自动态视频，为跨模态指令提供对齐监督。然后，我们提出TV-Edit，一个文本-视觉指令统一编辑框架，将基于拖拽或点的视觉指令与图像-文本语义上下文化，并将其提升为语义感知的控制表示，用于预训练的编辑骨干网络。通过整合语义意图和空间约束，TV-Edit相比纯文本或纯拖拽方法实现了更精确的空间控制、更少的指令歧义和更强的结构一致性。最后，我们建立了TV-Edit-Bench，一个精心设计的基准，用于评估语义忠实度、空间对齐和视觉一致性，通过地面真实参考和受控的文本-视觉变化进行可靠评估。我们在多个编辑骨干网络上的实验表明，TV-Edit始终产生更精确且忠实于意图的编辑，显著优于最先进的基于指令和基于拖拽的基线方法。
英文摘要：
Existing image editing methods can be generally categorized into textual instruction-based and visual prompt-based ones. Textual instructions are semantically expressive, but are limited by the coarse granularity of spatial control of the editing results. In contrast, visual prompts such as drag and point can provide precise spatial guidance, but are limited by the inherent ambiguity in semantic intent. To unify the strength of textual and visual prompts, we present Text-Vision Co-Instructed Image Editing, which jointly models textual instructions as semantic intent and sparse visual instructions as spatial guidance, aiming to achieve precise and intent-faithful image manipulation. To this end, we first construct a textual-visual instruction paired dataset with more than 23K samples derived from dynamic videos, enabling aligned supervision for cross-modal instruction. We then propose TV-Edit, a Textual-Visual instruction unified Editing framework to contextualize drag or point-based visual instructions with image-text semantics and lift them into semantic-aware control representations for pretrained editing backbones. By integrating semantic intent and spatial constraints, TV-Edit leads to more precise spatial control, less instruction ambiguity, and stronger structural consistency than text-only or drag-based alternatives. Finally, we establish TV-Edit-Bench, a deliberately designed benchmark to evaluate semantic faithfulness, spatial alignment, and visual consistency with ground-truth references and controlled textual-visual variations for reliable assessment. Our experiments across multiple editing backbones demonstrate that TV-Edit consistently yields more precise and intent-faithful edits, significantly outperforming state-of-the-art instruction-based and drag-based baselines.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
