In this paper, we propose an embarrassingly simple yet highly effective zero-shot semantic segmentation (ZS3) method, based on the pre-trained vision-language model CLIP. First, our study provides a couple of key discoveries: (i) the global tokens (a.k.a [CLS] tokens in Transformer) of the text branch in CLIP provide a powerful representation of semantic information and (ii) these text-side [CLS] tokens can be regarded as category priors to guide CLIP visual encoder pay more attention on the corresponding region of interest. Based on that, we build upon the CLIP model as a backbone which we extend with a One-Way [CLS] token navigation from text to the visual branch that enables zero-shot dense prediction, dubbed \textbf{ClsCLIP}. Specifically, we use the [CLS] token output from the text branch, as an auxiliary semantic prompt, to replace the [CLS] token in shallow layers of the ViT-based visual encoder. This one-way navigation embeds such global category prior earlier and thus promotes semantic segmentation. Furthermore, to better segment tiny objects in ZS3, we further enhance ClsCLIP with a local zoom-in strategy, which employs a region proposal pre-processing and we get ClsCLIP+. Extensive experiments demonstrate that our proposed ZS3 method achieves a SOTA performance, and it is even comparable with those few-shot semantic segmentation methods.
Cross-Modal Action Recognition in Egocentric Video Using Mamba: Integrating RGB and Hand Skeleton Streams via CLS Token Fusion Strategies
Univ. of Buenos Aires. Faculty of Exact and Natural Sciences. Dept. of Computer Science (DC)(布宜诺斯艾利斯大学。精确与自然科学学院。计算机科学系);CONICET-Univ. of Buenos Aires. Institute of Computer Sciences (ICC)(布宜诺斯艾利斯大学CONICET联合体。计算机科学研究所)
Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Models
School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Institution of Automation, Chinese Academy of Sciences(中国科学院自动化研究所)
Adversarial Robustness in Zero-Shot Learning:An Empirical Study on Class and Concept-Level Vulnerabilities
iFLYTEK Co., Ltd.(iFLYTEK公司);UCAS-Terminus AI Lab, University of Chinese Academy of Sciences(中国科学院大学Terminus AI实验室);School of Computer Science, the University of Nottingham(诺丁汉大学计算机学院);TGAI lab, the Westlake University(西湖大学TGAI实验室);RippleInfo Co., Ltd(RippleInfo公司)
TokenCLIP: Token-wise Prompt Learning for Zero-shot Anomaly Detection
Zhejiang University(浙江大学);Tencent YouTu Lab(腾讯YouTu实验室);Singapore Management University(新加坡管理学院);Qilu University of Technology (Shandong Academy of Sciences)(青岛科技大学(山东科学院))
Beyond Class Tokens: LLM-guided Dominant Property Mining for Few-shot Classification
School of Artificial Intelligence and the National Engineering Laboratory of Big Data System Computing Technology, Shenzhen University(人工智能学院和大数据系统计算技术国家工程实验室,深圳大学);School of Biomedical Engineering, Shenzhen University Medical School(生物医学工程学院,深圳大学医学院)
Zero-shot sketch-based remote sensing image retrieval based on multi-level and attention-guided tokenization
Anhui Province Key Laboratory of Wetland Ecosystem Protection and Restoration, Anhui University(安徽大学安徽省湿地生态保护与修复重点实验室);School of Resources and Environmental Engineering, Anhui University(安徽大学资源与环境工程学院);Shanghai Ubiquitous Navigation Technology Co. Ltd(上海普适导航科技股份有限公司);The 54th Research Institute of China Electronics Technology Group Corporation(中国电子科技集团公司第五十四研究所)
Improving CLIP Adaptation by Breaking Tail Alignment for Source-Free Cross-Domain Few-Shot Learning
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China(华中科技大学计算机科学与技术学院);Institute of Artificial Intelligence, Huazhong University of Science and Technology, Wuhan, China(华中科技大学人工智能研究院)
Rethinking Position Embedding as a Context Controller for Multi-Reference and Multi-Shot Video Generation
Wuhan University(武汉大学);University of Science and Technology of China(中国科学技术大学);Hong Kong University of Science and Technology(香港科技大学);Tsinghua University(清华大学)
School of Computer Science and Engineering at South China University of Technology(华南理工大学计算机科学与工程学院);School of Intelligence Science and Engineering, Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳)智能科学与工程学院);School of Computing and Information Systems, Singapore Management University(新加坡管理学院计算机与信息系)
Few-shot multi-token DreamBooth with LoRa for style-consistent character generation
Institute of Smart Cities (ISC)(智能城市研究所);Department of Statistics, Computer Science and Mathematics(统计学、计算机科学与数学系);Public University of Navarre (UPNA)(纳瓦拉公共大学)
PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classification
Dalian University of Technology(大连理工大学);School of Computer Science and Technology(计算机科学与技术学院);School of Computer Science and Engineering(计算机科学与工程学院);Dalian Minzu University(大连民族大学);School of Information and Communication Engineering(信息与通信工程学院)