Referring segmentation aims to segment a target object related to a natural language expression. Key challenges of this task are understanding the meaning of complex and ambiguous language expressions and determining the relevant regions in the image with multiple objects by referring to the expression. Recent models have focused on the early fusion with the language features at the intermediate stage of the vision encoder, but these approaches have a limitation that the language features cannot refer to the visual information. To address this issue, this paper proposes a novel architecture, Cross-aware early fusion with stage-divided Vision and Language Transformer encoders (CrossVLT), which allows both language and vision encoders to perform the early fusion for improving the ability of the cross-modal context modeling. Unlike previous methods, our method enables the vision and language features to refer to each other's information at each stage to mutually enhance the robustness of both encoders. Furthermore, unlike the conventional scheme that relies solely on the high-level features for the cross-modal alignment, we introduce a feature-based alignment scheme that enables the low-level to high-level features of the vision and language encoders to engage in the cross-modal alignment. By aligning the intermediate cross-modal features in all encoder stages, this scheme leads to effective cross-modal fusion. In this way, the proposed approach is simple but effective for referring image segmentation, and it outperforms the previous state-of-the-art methods on three public benchmarks.
Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentation
Sun Yat-sen University(中山大学);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳));Shenzhen Research Institute of Big Data(深圳大数据研究院);AI Institute, Fudan University(复旦大学人工智能研究院);Tencent AI Lab(腾讯人工智能实验室)
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
University of Oxford(牛津大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua-Berkeley Shenzhen Institute, Tsinghua University(清华大学清华-伯克利深圳学院);SenseTime Research(商汤科技研究院);The University of Hong Kong(香港大学)
EviRCOD: Evidence-Guided Probabilistic Decoding for Referring Camouflaged Object Detection
West China Hospital, Sichuan University(四川大学华西医院);Independent Researcher(独立研究员);Department of Data Science and Artificial Intelligence, Auckland University of Technology(奥克兰理工大学数据科学与人工智能系)
Semantics-Aware Dynamic Localization and Refinement for Referring Image Segmentation
University of Oxford(牛津大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua-Berkeley Shenzhen Institute, Tsinghua University(清华大学深圳国际研究生院(清华-伯克利深圳学院));The University of Hong Kong(香港大学)
PTQ4RIS: Post-Training Quantization for Referring Image Segmentation
School of Electronic and Electrical Engineering, Shanghai University of Engineering Science(上海工程技术大学电子与电气工程学院);SenseTime(商汤科技);Carnegie Mellon University(卡内基梅隆大学)
Intra- and Cross-Scale Feature Interaction Network for Referring Remote Sensing Image Segmentation
School of Artificial Intelligence, Optics and Electronics (iOPEN), Northwestern Polytechnical University(人工智能、光学与电子学院(iOPEN),西北工业大学);School of Computer Science, Northwestern Polytechnical University(计算机科学学院,西北工业大学);department of Software, Northwest Institute of Nuclear Technology(软件系,西北核技术研究院)
Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation
Beihang University(北京航空航天大学);Meituan(美团);Hangzhou Innovation Institute, Beihang University(北京航空航天大学杭州创新研究院);Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);University of Chinese Academy of Sciences(中国科学院大学)
Location-Aware Pretraining for Medical Difference Visual Question Answering
Department of Electrical and Computer Engineering, Carnegie Mellon University, Kigali, Rwanda(电气与计算机工程系,卡内基梅隆大学,刚果(金)基利齐);University of Melbourne, Melbourne, Australia(墨尔本大学,墨尔本,澳大利亚)
SSP-SAM: SAM with Semantic-Spatial Prompt for Referring Expression Segmentation
School of Computer Science and Engineering, Nanjing University of Science and Technology(南京理工大学计算机科学与工程学院);NExT++ Lab, School of Computing, National University of Singapore(新加坡国立大学计算机学院NExT++实验室);Institute of Computing Technology, Chinese Academy of Sciences(中国科学院计算技术研究所)
Uncertainty Quantification in Medical Image Segmentation with Multi-decoder U-Net
Harbin Institute of Technology at Shenzhen(哈尔滨工业大学(深圳));Peng Cheng Lab(鹏城实验室);MindsGo Co.Ltd(MindsGo 有限公司);Capital Medical University(首都医科大学);Xuanwu Hospital Capital Medical University(首都医科大学宣武医院)
MPG-SAM 2: Adapting SAM 2 with Mask Priors and Global Context for Referring Video Object Segmentation
National Engineering Research Center for Multimedia Software, School of Computer Science, Wuhan University(国家多媒体软件工程研究中心,计算机学院,武汉大学);Hong Kong University of Science and Technology(香港科学与技术大学);Horizon Robotics
The Marine Debris Dataset for Forward-Looking Sonar Semantic Segmentation
Netaji Subhas Institute Of Technology(Netaji Subhas理工学院);German Research Center for Artificial Intelligence(德国人工智能研究中心)
海洋废弃物的准确检测和分割对于保持水体清洁至关重要。本文提出了一种新的数据集海洋垃圾分割收集使用前视声纳(FLS)。该数据集由使用ARIS Explorer 3000传感器捕获的1868幅FLS图像组成。用于生成该数据集的对象包括典型的家用海洋废弃物和干扰海洋对象(轮胎,钩子,阀门等),分为11类加上一个背景类。性能的最先进的语义分割架构与各种编码器已在此数据集上进行了分析,并作为基线结果。由于图像是灰度的,因此没有使用预训练的权重。使用Intersection over Union(IoU)进行比较。性能最好的模型是具有ResNet34主干的Unet,其容量为0.7481 mIoU。该数据集可在www.example.com上获得
RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics
School of Software, Beihang University(北京航空航天大学软件学院);State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University(北京大学计算机学院);Beijing Academy of Artificial Intelligence(北京人工智能研究院)
Self-paced Multi-grained Cross-modal Interaction Modeling for Referring Expression Comprehension
School of Software Technology, Zhejiang University(浙江大学软件学院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院);Zhejiang University-University of Illinois at Urbana-Champaign Institute, Zhejiang University(浙江大学-伊利诺伊大学厄巴纳-香槟分校研究院)
Blind to Position, Biased in Language: Probing Mid-Layer Representational Bias in Vision-Language Encoders for Zero-Shot Language-Grounded Spatial Understanding
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院);Samsung Electronics(三星电子)
Wuhan University(武汉大学);Insta360 Research(Insta360研究院);Department of EECS, University of California, Merced(加州大学默塞德分校电子工程与计算机科学系);Nanyang Technological University(南洋理工大学);Institute of Automation of the Chinese Academy of Sciences(中国科学院自动化研究所)
Referring Video Object Segmentation with Cross-Modality Proxy Queries
DUT-RU International School of Information Science & Engineering, Dalian University of Technology, China(大连理工大学国际信息科学与工程学院,中国);Chinese University of Hong Kong, China(香港中文大学,中国);University of Sydney, Australia(悉尼大学,澳大利亚)