arXiv 2605.279242026-05-27cs.CV

SIGMA: 基于语义差异的指令引导掩码标注器用于文本驱动图像操作定位

SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization

Shenzhen Campus of Sun Yat-sen University（中山大学深圳校区）；Guangdong Provincial Key Laboratory of Intelligent Information Processing and Shenzhen Key Laboratory of Media Security（广东省智能信息处理重点实验室和深圳媒体安全重点实验室）；Shenzhen University of Advanced Technology and Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术大学和深圳先进技术研究所，中国科学院）；Alibaba Group（阿里巴巴集团）；Shenzhen MSU-BIT University（深圳MSU-BIT大学）

Peiyu Zhuang, Jianquan Yang, Haodong Li, Zhuoying Cai, Ruitao Xie, Jishen Zeng, Baoying Chen, Jiwu Huang, Xiaochun Cao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出SIGMA方法，通过视觉基础模型中的语义特征差异和指令引导的空间先验，自动从公开编辑数据集中生成像素级掩码，用于训练图像操作定位模型，在五个基准上F1提升12.20%，并生成约110万训练集使六个检测器平均F1提升18.34%。

摘要：

文本驱动的图像编辑发展迅速，但可靠地定位这些操作需要在大规模像素标注数据集上训练的图像操作定位（IML）模型，目前尚无低成本获取此类训练数据的方法。我们观察到这些数据实际上已经以伪装形式存在：公开编辑数据集包含数百万个与IML训练样本结构相同的（原始、编辑）图像对，仅缺少像素级掩码。自动恢复这些掩码并非易事：像素差异被扩散引起的所有像素扰动淹没，而仅基于指令的定位只能定位提示描述的内容，遗漏了意外的编辑副作用。我们提出SIGMA（语义差异指令引导掩码标注器），它在视觉基础骨干网络中进行语义特征差异计算，并通过双向跨模态精炼将指令导出的空间先验注入视觉流，在编辑器忠实实现用户意图时放大预期编辑区域的差异信号。SIGMA通过两个互补阶段训练：第一阶段在修复掩码上进行监督；第二阶段通过VAE往返噪声校准、EMA自训练和编辑噪声解耦损失来弥合扩散域偏移。SIGMA在五个基准上优于现有自动掩码生成器（F1提升12.20%，IoU提升11.16%）。当应用于公开编辑语料库时，它生成了约110万IML训练集，使六个不同检测器在五个数据集上平均F1提升18.34%，将以前未使用的编辑数据转化为IML的模型无关监督资源。论文被接收后我们将立即发布完整代码库。

英文摘要：

Text-driven image editing has advanced rapidly, but reliably localizing these manipulations requires image manipulation localization (IML) models trained on large pixel-annotated datasets, and there is still no low-cost way to obtain such training data at scale. We observe that these data already exist in disguise: public editing datasets contain millions of structurally identical (original, edited) pairs to IML training samples, lacking only pixel-level masks. Recovering these masks automatically is non-trivial: pixel differencing is overwhelmed by diffusion-induced perturbations across all pixels, and instruction-only grounding localizes only what the prompt describes, missing unintended editor side-effects. We propose SIGMA (Semantic-difference Instruction-Grounding Mask Annotator), which performs semantic-feature differencing in a vision foundation backbone and injects an instruction-derived spatial prior into this visual stream via bidirectional cross-modal refinement, amplifying the difference signal at intended-edit regions when the editor faithfully realizes user intent. SIGMA is trained in two complementary stages: Stage I supervises on inpainting masks; Stage II closes the diffusion-domain shift via VAE-roundtrip noise calibration, EMA self-training, and an edit-noise disentanglement loss. SIGMA outperforms existing automatic mask generators on five benchmarks (+12.20% F1, +11.16% IoU). When applied to public editing corpora, it produces a ~1.1M IML training set that improves six diverse detectors by +18.34% F1 across five datasets, turning previously unused editing data into a model-agnostic supervisory resource for IML. We'll release the full codebase as soon as the paper is accepted.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。