From Pixels to Concepts: Do Segmentation Models Understand What They Segment?
Department of Electrical and Computer Engineering, The University of Hong Kong(香港大学电子与计算机工程系);School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);CASIC, The University of Hong Kong(香港大学中国科学院自动化所)
Shuang Liang, Zeqing Wang, Yuxian Li, Xihui Liu, Han Wang
分割是一种基础的视觉任务,支撑着众多下游应用。最近的可提示分割模型,如Segment Anything Model 3(SAM3),将分割从类别无关的掩码预测扩展到基于高级文本提示的概念引导定位。然而,现有基准主要评估掩码准确度或物体存在性,不清楚这些模型是否真正理解所分割的概念,还是依赖视觉显著但语义误导的线索。我们引入CAFE:Counterfactual Attribute Factuality Evaluation,一个评估可提示分割模型概念忠实性的新基准。我们的CAFE基于属性层面的反事实操纵:目标区域和真实掩码被保留,而表面外观、上下文或材料组成等属性被修改以引入误导性语义线索。该基准包含2,146对测试样本,每对样本包括目标图像、真实掩码、正提示和误导性负提示。这些样本覆盖三个反事实类别:Superficial Mimicry(SM)、Context Conflict(CC)和Ontological Conflict(OC)。我们评估了各种模型类型和大小在CAFE上的表现。实验揭示了定位质量与概念判别之间的系统性差距:模型在误导性提示下仍能生成准确的掩码,表明强大的掩码预测并不必然意味着语义根基的忠实。CAFE提供了一个受控的基准,用于诊断可提示分割模型是否执行概念忠实的根基,而非基于捷径的掩码检索。
英文摘要:
Segmentation is a fundamental vision task underlying numerous downstream applications. Recent promptable segmentation models, such as Segment Anything Model 3 (SAM3), extend segmentation from category-agnostic mask prediction to concept-guided localization conditioned on high-level textual prompts. However, existing benchmarks primarily evaluate mask accuracy or object presence, leaving unclear whether these models faithfully ground the queried concept or instead rely on visually salient but semantically misleading cues. We introduce CAFE: \textbf{C}ounterfactual \textbf{A}ttribute \textbf{F}actuality \textbf{E}valuation, a novel benchmark for evaluating concept-faithful segmentation in promptable segmentation models. Our \textbf{CAFE} is built on attribute-level counterfactual manipulation: the target region and ground-truth mask are preserved, while attributes such as surface appearance, context, or material composition are modified to introduce misleading semantic cues. The benchmark contains 2,146 paired test samples, each consisting of a target image, a ground-truth mask, a positive prompt, and a misleading negative prompt. These samples cover three counterfactual categories: Superficial Mimicry (\textbf{SM}), Context Conflict (\textbf{CC}), and Ontological Conflict (\textbf{OC}). We evaluate various model types and sizes on our CAFE. Experiments reveal a systematic gap between localization quality and concept discrimination: models often generate accurate masks even for misleading prompts, suggesting that strong mask prediction does not necessarily imply faithful semantic grounding. Our CAFE provides a controlled benchmark for diagnosing whether promptable segmentation models perform concept-faithful grounding rather than shortcut-driven mask retrieval.
Semantic Segmentation of highly class imbalanced fully labelled 3D volumetric biomedical images and unsupervised Domain Adaptation of the pre-trained Segmentation Network to segment another fully unlabelled Biomedical 3D Image stack
Cross-Image Relational Knowledge Distillation for Semantic Segmentation
Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);Horizon Robotics(地平线机器人);Wuhan University(武汉大学)
Retinal Vessel Segmentation with Pixel-wise Adaptive Filters
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合性国家科学中心人工智能研究院)
Regression and Learning with Pixel-wise Attention for Retinal Fundus Glaucoma Segmentation and Detection
University of Florida(佛罗里达大学)
眼科医生观察视网膜眼底图像是青光眼的主要诊断方法。然而,仅通过人工观察仍然难以区分病变的特征,特别是在青光眼早期。在本文中,我们提出了两种基于深度学习的自动算法,用于青光眼检测和视盘和视杯分割。我们利用注意力机制来学习像素级特征以进行准确预测。特别是,我们提出了两个卷积神经网络,可以专注于学习各种像素级特征。此外,我们还开发了几种注意力策略来指导网络学习对预测准确性有重大影响的重要特征。我们在验证数据集上评估了我们的方法,提出的两个任务的解决方案可以取得令人印象深刻的结果,并优于当前最先进的方法。\textit{The code is available at \url{https://github.com/cswin/RLPA}}。
ConnNet: A Long-Range Relation-Aware Pixel-Connectivity Network for Salient Segmentation
UiT The Arctic University of Norway(挪威北极大学);Carnegie Mellon University(卡内基梅隆大学);Cornell University(康奈尔大学);Petuum Inc.(派拓姆公司);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学)
PSGR: Pixel-wise Sparse Graph Reasoning for COVID-19 Pneumonia Segmentation in CT Images
Northwestern Polytechnical University(西北工业大学);Intel Labs(英特尔实验室);The University of Sydney(悉尼大学);University of Pittsburgh(匹兹堡大学);JD Finance America Corporation(京东金融美国公司)
Class-independent sequential full image segmentation, using a convolutional net that finds a segment within an attention region, given a pointer pixel within this segment
SegFix: Model-Agnostic Boundary Refinement for Segmentation
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS), Institute of Computing Technology, CAS(中国科学院计算技术研究所智能信息处理重点实验室);University of Chinese Academy of Sciences(中国科学院大学);Microsoft Research Asia(微软亚洲研究院);University of Science and Technology of China(中国科学技术大学)
Unsupervised Domain Adaptation in Semantic Segmentation Based on Pixel Alignment and Self-Training
Peking University(北京大学);National Engineering Laboratory for Big Data Analysis and Applications(大数据分析与应用国家工程实验室);Center for Data Science(数据科学中心);Beijing International Center for Mathematical Research (BICMR)(北京国际数学研究中心);Center for Data Science in Health and Medicine(健康医疗数据科学中心)
RPM-Net: Robust Pixel-Level Matching Networks for Self-Supervised Video Object Segmentation
KAIST(韩国科学技术院)
本文介绍了一种无需人工标记数据的自监督视频对象分割方法,具体而言,我们提出了鲁棒像素级匹配网络(RPM-Net),这是一种新的深度架构,仅使用来自未标记视频的颜色信息进行训练,匹配相邻帧之间的像素。从技术上讲,RPM-Net可以分为两个主要模块。嵌入模块首先将输入图像投影到高维嵌入空间。与以往的基于可变形卷积的方法不同,本文的匹配模块采用可变形卷积来关注时空相邻像素中的相似特征,实验结果表明,选择性特征采样提高了视频对象分割对相机抖动等挑战性问题的鲁棒性,快速运动、变形和遮挡。此外,我们在三个公共数据集(即,DAVIS-2017、SegTrack-v2和Youtube-Objects),并在自监督视频对象分割方面实现最先进的性能。此外,我们显着减少了自监督和全监督视频对象分割之间的性能差距(DAVIS-2017验证集上为41.0% vs. 52.5%)
FGAseg: Fine-Grained Pixel-Text Alignment for Open-Vocabulary Semantic Segmentation
University of Science and Technology of China(中国科学技术大学);Institute of Artificial Intelligence (TeleAI), China Telecom(中国电信人工智能研究院(TeleAI));School of Artificial Intelligence, OPtics and ElectroNics (iOPEN), Northwestern Polytechnical University(西北工业大学人工智能、光学与电子学院(iOPEN))
Learning Category- and Instance-Aware Pixel Embedding for Fast Panoptic Segmentation
Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);University of Chinese Academy of Sciences (UCAS)(中国科学院大学);Horizon Robotics(地平线机器人);CAS Center for Excellence in Brain Science and Intelligence Technology(中国科学院脑科学与智能技术卓越创新中心)