Referring Expression Segmentation (RES) aims to provide a segmentation mask of the target object in an image referred to by the text (i.e., referring expression). Existing methods require large-scale mask annotations. Moreover, such approaches do not generalize well to unseen/zero-shot scenarios. To address the aforementioned issues, we propose a weakly-supervised bootstrapping architecture for RES with several new algorithmic innovations. To the best of our knowledge, ours is the first approach that considers only a fraction of both mask and box annotations (shown in Figure 1 and Table 1) for training. To enable principled training of models in such low-annotation settings, improve image-text region-level alignment, and further enhance spatial localization of the target object in the image, we propose Cross-modal Fusion with Attention Consistency module. For automatic pseudo-labeling of unlabeled samples, we introduce a novel Mask Validity Filtering routine based on a spatially aware zero-shot proposal scoring approach. Extensive experiments show that with just 30% annotations, our model SafaRi achieves 59.31 and 48.26 mIoUs as compared to 58.93 and 48.19 mIoUs obtained by the fully-supervised SOTA method SeqTR respectively on RefCOCO+@testA and RefCOCO+testB datasets. SafaRi also outperforms SeqTR by 11.7% (on RefCOCO+testA) and 19.6% (on RefCOCO+testB) in a fully-supervised setting and demonstrates strong generalization capabilities in unseen/zero-shot tasks.
Entity-enhanced Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding
University of Chinese Academy of Sciences(中国科学院大学);Peng Cheng Laboratory(鹏城实验室);University of Science and Technology of China(中国科学技术大学);Nanjing University of Science and Technology(南京理工大学);Huawei Cloud & AI(华为云与人工智能部门)
Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS(中国科学院计算技术研究所智能信息处理重点实验室);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学);Peng Cheng Laboratory(鹏城实验室)
Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding
Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学);University of Science and Technology of China(中国科学技术大学)
The University of Western Australia(西澳大利亚大学);Murdoch University(莫道克大学);Sun Yat-sen University(中山大学);The Hong Kong University of Science and Technology(香港科技大学)
WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation
The University of Hong Kong(香港大学);King’s College London(伦敦国王学院);Huazhong Agricultural University(华中农业大学);National University of Singapore(新加坡国立大学);Shanghai AI Laboratory(上海人工智能实验室)
Understanding What Is Not Said:Referring Remote Sensing Image Segmentation with Scarce Expressions
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China(教育部多媒体可信感知与高效计算重点实验室);Institute of Artificial Intelligence, Xiamen University(厦门大学人工智能研究院);Xiamen University(厦门大学);School of Informatics, Xiamen University(厦门大学信息学院)
BigEarthNet.txt: A Large-Scale Multi-Sensor Image-Text Dataset and Benchmark for Earth Observation
BIFOLD(柏林智能数据与机器学习研究所);Technische Universität Berlin(柏林工业大学);University of Trento(特伦托大学);National Technical University(雅典国家技术大学);National Observatory of Athens(雅典国家天文台)
Training-Free Open-Vocabulary Visual Grounding for Remote Sensing Images and Videos
School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);Interdisciplinary Institute of Artificial Intelligence, Xidian University(西安电子科技大学跨学科人工智能研究院);School of Artificial Intelligence, Xidian University(西安电子科技大学人工智能学院);Northwest Institute of Nuclear Technology(西北核技术研究所);School of Physics and Information Engineering, Fuzhou University(福州大学物理与信息工程学院)
Beyond Point Annotation: A Weakly Supervised Network Guided by Multi-Level Labels Generated from Four-Point Annotation for Thyroid Nodule Segmentation in Ultrasound Image
Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China(机器人科学与工程学院,东北大学,沈阳,中国);Division of Biomedical Engineering, University of Saskatchewan, Saskatoon, Canada(生物医学工程系,萨斯喀彻温大学,萨斯喀彻温,加拿大);Department of Ultrasound, Shengjing Hospital of China Medical University, Shenyang, China(超声科,中国医科大学附属盛京医院,沈阳,中国)
From Text to Mask: Localizing Entities Using the Attention of Text-to-Image Diffusion Models
Institute for Artificial Intelligence, Tsinghua University(清华大学人工智能研究院);Beijing National Research Center for Information Science and Technology(北京信息科学与技术国家研究中心);Algorithm Research, Aibee Inc.(爱笔科技算法研究部)