arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2605.09591cs.CV
收藏

从像素到概念:分割模型是否理解它们所分割的内容?

From Pixels to Concepts: Do Segmentation Models Understand What They Segment?

Department of Electrical and Computer Engineering, The University of Hong Kong(香港大学电子与计算机工程系);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);CASIC, The University of Hong Kong(香港大学中国科学院自动化所)

Shuang Liang, Zeqing Wang, Yuxian Li, Xihui Liu, Han Wang

AI总结:

本文提出CAFE基准,评估可提示分割模型对概念的忠实性。通过属性层面的反事实操纵,测试模型在误导性提示下的表现,揭示分割模型可能依赖视觉显著但语义误导的线索。

摘要:

分割是一种基础的视觉任务,支撑着众多下游应用。最近的可提示分割模型,如Segment Anything Model 3(SAM3),将分割从类别无关的掩码预测扩展到基于高级文本提示的概念引导定位。然而,现有基准主要评估掩码准确度或物体存在性,不清楚这些模型是否真正理解所分割的概念,还是依赖视觉显著但语义误导的线索。我们引入CAFE:Counterfactual Attribute Factuality Evaluation,一个评估可提示分割模型概念忠实性的新基准。我们的CAFE基于属性层面的反事实操纵:目标区域和真实掩码被保留,而表面外观、上下文或材料组成等属性被修改以引入误导性语义线索。该基准包含2,146对测试样本,每对样本包括目标图像、真实掩码、正提示和误导性负提示。这些样本覆盖三个反事实类别:Superficial Mimicry(SM)、Context Conflict(CC)和Ontological Conflict(OC)。我们评估了各种模型类型和大小在CAFE上的表现。实验揭示了定位质量与概念判别之间的系统性差距:模型在误导性提示下仍能生成准确的掩码,表明强大的掩码预测并不必然意味着语义根基的忠实。CAFE提供了一个受控的基准,用于诊断可提示分割模型是否执行概念忠实的根基,而非基于捷径的掩码检索。

英文摘要:

Segmentation is a fundamental vision task underlying numerous downstream applications. Recent promptable segmentation models, such as Segment Anything Model 3 (SAM3), extend segmentation from category-agnostic mask prediction to concept-guided localization conditioned on high-level textual prompts. However, existing benchmarks primarily evaluate mask accuracy or object presence, leaving unclear whether these models faithfully ground the queried concept or instead rely on visually salient but semantically misleading cues. We introduce CAFE: \textbf{C}ounterfactual \textbf{A}ttribute \textbf{F}actuality \textbf{E}valuation, a novel benchmark for evaluating concept-faithful segmentation in promptable segmentation models. Our \textbf{CAFE} is built on attribute-level counterfactual manipulation: the target region and ground-truth mask are preserved, while attributes such as surface appearance, context, or material composition are modified to introduce misleading semantic cues. The benchmark contains 2,146 paired test samples, each consisting of a target image, a ground-truth mask, a positive prompt, and a misleading negative prompt. These samples cover three counterfactual categories: Superficial Mimicry (\textbf{SM}), Context Conflict (\textbf{CC}), and Ontological Conflict (\textbf{OC}). We evaluate various model types and sizes on our CAFE. Experiments reveal a systematic gap between localization quality and concept discrimination: models often generate accurate masks even for misleading prompts, suggesting that strong mask prediction does not necessarily imply faithful semantic grounding. Our CAFE provides a controlled benchmark for diagnosing whether promptable segmentation models perform concept-faithful grounding rather than shortcut-driven mask retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑