Calibration & Reconstruction: Deep Integrated Language for Referring Image Segmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
Yichen Yan, Xingjian He, Sihan Chen, Jing Liu
备注:
9 pages, 8 figures ICMR2024. arXiv admin note: text overlap with arXiv:2305.14969
Referring image segmentation aims to segment an object referred to by natural language expression from an image. The primary challenge lies in the efficient propagation of fine-grained semantic information from textual features to visual features. Many recent works utilize a Transformer to address this challenge. However, conventional transformer decoders can distort linguistic information with deeper layers, leading to suboptimal results. In this paper, we introduce CRFormer, a model that iteratively calibrates multi-modal features in the transformer decoder. We start by generating language queries using vision features, emphasizing different aspects of the input language. Then, we propose a novel Calibration Decoder (CDec) wherein the multi-modal features can iteratively calibrated by the input language features. In the Calibration Decoder, we use the output of each decoder layer and the original language features to generate new queries for continuous calibration, which gradually updates the language features. Based on CDec, we introduce a Language Reconstruction Module and a reconstruction loss. This module leverages queries from the final layer of the decoder to reconstruct the input language and compute the reconstruction loss. This can further prevent the language information from being lost or distorted. Our experiments consistently show the superior performance of our approach across RefCOCO, RefCOCO+, and G-Ref datasets compared to state-of-the-art methods.
Cross-Modal Progressive Comprehension for Referring Segmentation
Beihang University(北京航空航天大学);Chinese Academy of Sciences(中国科学院);University of Chinese Academy of Sciences(中国科学院大学);Beijing Jiaotong University(北京交通大学);Sun Yat-sen University(中山大学);Pazhou Lab(琶洲实验室)
Understanding What Is Not Said:Referring Remote Sensing Image Segmentation with Scarce Expressions
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能研究院);Xiamen University(厦门大学);School of Informatics, Xiamen University(厦门大学信息学院)
Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
School of Computer Science and Artificial Intelligence, Wuhan University of Technology(武汉理工大学计算机科学与人工智能学院);School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院);University of Electronic Science and Technology of China(电子科技大学);Wuhan Vocational College of Software and Engineering(武汉软件工程职业学院)
Referring Segmentation in Images and Videos with Cross-Modal Self-Attention Network
Wenzhou University(温州大学);University of Manitoba(曼尼托巴大学);Shanghai Institute for Advanced Communication and Data Science(上海先进通信与数据科学研究院);Shanghai University(上海大学)
OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Pengcheng Laboratory(鹏城实验室);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳))
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(中国教育部多媒体可信感知与高效计算重点实验室,厦门大学);Youtu Lab, Tencent, P.R. China(腾讯视频实验室,中国);OpenGVLab, Shanghai AI Laboratory(上海AI实验室)
CausalCLIPSeg: Unlocking CLIP's Potential in Referring Medical Image Segmentation with Causal Intervention
Wuhan University of Technology(武汉理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);MedAI Technology (Wuxi) Co. Ltd.(MedAI科技(无锡)有限公司);Technical University of Munich(慕尼黑工业大学)
University of Sydney(悉尼大学);OPPO Research Institute(OPPO研究院);Beijing University of Posts and Telecommunications(北京邮电大学);Kuaishou Technology(快手科技);University of Melbourne(墨尔本大学)
MMNet: Multi-Mask Network for Referring Image Segmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Science and Technology Beijing(北京科技大学);University of Chinese Academy of Sciences(中国科学院大学)
SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction
Shanghai Jiao Tong University(上海交通大学);Shanghai Innovation Institute(上海创新研究院);The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);University of Science and Technology of China(中国科学技术大学)
CMF: Cascaded Multi-model Fusion for Referring Image Segmentation
Beijing University of Posts and Telecommunications(北京邮电大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Artificial Intelligence Research, Chinese Academy of Sciences(中国科学院人工智能研究院);University of Chinese Academy of Sciences(中国科学院大学)
Don't Guess, Just Ask: Resolving Ambiguity in Referring Segmentation via Multi-turn Clarification
School of Computer Science and Engineering, Sun Yat-sen University(中山大学计算机科学与工程学院);Guangdong Province Key Laboratory of Information Security Technology(广东省信息安全技术重点实验室);Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education(教育部机器智能与先进计算重点实验室)
Video-Instrument Synergistic Network for Referring Video Instrument Segmentation in Robotic Surgery
Hong Kong University of Science and Technology (Guangzhou)(香港科技大学(广州));Imperial College London(帝国理工学院);Hong Kong University of Science and Technology(香港科技大学);The General Hospital of Southern Theatre Command of PLA(中国人民解放军南部战区总医院);National University of Singapore(新加坡国立大学)
Exploring Fine-Grained Image-Text Alignment for Referring Remote Sensing Image Segmentation
School of Information Science and Technology, Southwest Jiaotong University(西南交通大学信息科学与技术学院);Ant Group(蚂蚁集团);Luoyang Institute of Electro-optical Equipment, AVIC(中国航空工业集团公司洛阳电光设备研究所);Faculty of Geosciences and Engineering, Southwest Jiaotong University(西南交通大学地球科学与环境工程学院)