EAVL: Explicitly Align Vision and Language for Referring Image Segmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Science(中国科学院大学人工智能学院)
Yichen Yan, Xingjian He, Wenxuan Wang, Sihan Chen, Jing Liu
备注:
10 pages, 4 figures. arXiv admin note: text overlap with arXiv:2305.14969
Referring image segmentation (RIS) aims to segment an object mentioned in natural language from an image. The main challenge is text-to-pixel fine-grained correlation. In the previous methods, the final results are obtained by convolutions with a fixed kernel, which follows a similar pattern as traditional image segmentation. These methods lack explicit alignment of language and vision features in the segmentation stage, resulting in suboptimal correlation. In this paper, we introduce EAVL, a method explicitly aligning vision and language features. In contrast to fixed convolution kernels, we introduce a Vision-Language Aligner that aligns features in the segmentation stage using dynamic convolution kernels based on the input image and sentence. Specifically, we generate multiple queries representing different emphases of language expression. These queries are transformed into a series of query-based convolution kernels, which are applied in the segmentation stage to produce a series of masks. The final result is obtained by aggregating all masks. Our method harnesses the potential of the multi-modal features in the segmentation stage and aligns language features of different emphases with image features to achieve fine-grained text-to-pixel correlation. We surpass previous state-of-the-art methods on RefCOCO, RefCOCO+, and G-Ref by large margins. Additionally, our method is designed to be a generic plug-and-play module for cross-modality alignment in RIS task, making it easy to integrate with other RIS models for substantial performance improvements.
Exploring Fine-Grained Image-Text Alignment for Referring Remote Sensing Image Segmentation
School of Information Science and Technology, Southwest Jiaotong University(西南交通大学信息科学与技术学院);Ant Group(蚂蚁集团);Luoyang Institute of Electro-optical Equipment, AVIC(中国航空工业集团公司洛阳电光设备研究所);Faculty of Geosciences and Engineering, Southwest Jiaotong University(西南交通大学地球科学与环境工程学院)
CADFormer: Fine-Grained Cross-modal Alignment and Decoding Transformer for Referring Remote Sensing Image Segmentation
School of Computer Science and Technology, Wuhan University of Science and Technology(武汉科技大学计算机科学与技术学院);School of Information Science and Engineering, Wuhan University of Science and Technology(武汉科技大学信息科学与工程学院)
EviRCOD: Evidence-Guided Probabilistic Decoding for Referring Camouflaged Object Detection
West China Hospital, Sichuan University(四川大学华西医院);Independent Researcher(独立研究员);Department of Data Science and Artificial Intelligence, Auckland University of Technology(奥克兰理工大学数据科学与人工智能系)
CM-MaskSD: Cross-Modality Masked Self-Distillation for Referring Image Segmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);School of Artificial Intelligence, University of Chinese Academy of Science(中国科学院大学人工智能学院);University of Science and Technology Beijing(北京科技大学)
University of Sydney(悉尼大学);OPPO Research Institute(OPPO研究院);Beijing University of Posts and Telecommunications(北京邮电大学);Kuaishou Technology(快手科技);University of Melbourne(墨尔本大学)
CausalCLIPSeg: Unlocking CLIP's Potential in Referring Medical Image Segmentation with Causal Intervention
Wuhan University of Technology(武汉理工大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);MedAI Technology (Wuxi) Co. Ltd.(MedAI科技(无锡)有限公司);Technical University of Munich(慕尼黑工业大学)
Progressive Prompt-Guided Cross-Modal Reasoning for Referring Image Segmentation
School of Computer Science and Artificial Intelligence, Wuhan University of Technology(武汉理工大学计算机科学与人工智能学院);School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院);University of Electronic Science and Technology of China(电子科技大学);Wuhan Vocational College of Software and Engineering(武汉软件工程职业学院)
CMF: Cascaded Multi-model Fusion for Referring Image Segmentation
Beijing University of Posts and Telecommunications(北京邮电大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Beijing Academy of Artificial Intelligence(北京人工智能研究院);Artificial Intelligence Research, Chinese Academy of Sciences(中国科学院人工智能研究院);University of Chinese Academy of Sciences(中国科学院大学)
Image-Conditioned Instance Prompt Network for Referring Remote Sensing Image Segmentation
School of Computer Science, Northwestern Polytechnical University, Xi'an, China(西北工业大学计算机科学学院,西安,中国);College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai, China(复旦大学智能机器人与先进制造学院,上海,中国);Shenzhen Research Institute of Northwestern Polytechnical University, Shenzhen, China(西北工业大学深圳研究院,深圳,中国)
SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction
Shanghai Jiao Tong University(上海交通大学);Shanghai Innovation Institute(上海创新研究院);The Chinese University of Hong Kong(香港中文大学);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Fudan University(复旦大学);University of Science and Technology of China(中国科学技术大学)
A Text-Image Fusion Method with Data Augmentation Capabilities for Referring Medical Image Segmentation
College of Information Science and Engineering(信息科学与工程学院);Ritsumeikan University(立命馆大学);Tiwaki Co., Ltd.(Tiwaki公司);Dalian University of Technology(大连理工大学);School of Software(软件学院);Zhejiang University(浙江大学)
LAVT: Language-Aware Vision Transformer for Referring Image Segmentation
University of Oxford(牛津大学);Shanghai AI Laboratory(上海人工智能实验室);Tsinghua-Berkeley Shenzhen Institute, Tsinghua University(清华大学清华-伯克利深圳学院);SenseTime Research(商汤科技研究院);The University of Hong Kong(香港大学)
Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention
AI Graduate Program, University of the Philippines(菲律宾大学人工智能研究生项目);EEEI, University of the Philippines(菲律宾大学电子工程与信息学院);Samsung R&D Institute Philippines(三星菲律宾研发研究所)
Referring Image Segmentation via Cross-Modal Progressive Comprehension
Institute of Information Engineering, Chinese Academy of Sciences(中国科学院信息工程研究所);School of Cyber Security, University of Chinese Academy of Sciences(中国科学院大学网络空间安全学院);School of Computer Science and Engineering, Beihang University(北京航空航天大学计算机学院);Sun Yat-sen University(中山大学);University of Technology Sydney(悉尼科技大学);AI Institute(360人工智能研究院)
RIS-LAD: A Benchmark and Model for Referring Low-Altitude Drone Image Segmentation
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能研究院);School of Informatics, Xiamen University(厦门大学信息学院)