Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting
Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);Linköping University(林雪平大学);University of Central Florida(中佛罗里达大学)
Syed Talal Wasim, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, Mubarak Shah
Adopting contrastive image-text pretrained models like CLIP towards video classification has gained attention due to its cost-effectiveness and competitive performance. However, recent works in this area face a trade-off. Finetuning the pretrained model to achieve strong supervised performance results in low zero-shot generalization. Similarly, freezing the backbone to retain zero-shot capability causes significant drop in supervised accuracy. Because of this, recent works in literature typically train separate models for supervised and zero-shot action recognition. In this work, we propose a multimodal prompt learning scheme that works to balance the supervised and zero-shot performance under a single unified training. Our prompting approach on the vision side caters for three aspects: 1) Global video-level prompts to model the data distribution; 2) Local frame-level prompts to provide per-frame discriminative conditioning; and 3) a summary prompt to extract a condensed video representation. Additionally, we define a prompting scheme on the text side to augment the textual context. Through this prompting scheme, we can achieve state-of-the-art zero-shot performance on Kinetics-600, HMDB51 and UCF101 while remaining competitive in the supervised setting. By keeping the pretrained backbone frozen, we optimize a much lower number of parameters and retain the existing general representation which helps achieve the strong zero-shot performance. Our codes/models are released at https://github.com/TalalWasim/Vita-CLIP.
STOP: Integrated Spatial-Temporal Dynamic Prompting for Video Understanding
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机研究所);Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);School of Artificial Intelligence and Automation, Huazhong University of Science and Technology(华中科技大学人工智能与自动化学院)
PSMP-CLIP: Patch-Prompt SAM and Multi-Semantic Prompting for CLIP-Based Zero-Shot Anomaly Detection
Harbin Engineering University(哈尔滨工程大学);Key Laboratory of Advanced Marine Communication and Information Technology(先进海洋通信与信息技术重点实验室);University of Toronto(多伦多大学);Kanagawa University(神奈川大学)
Improving Visual Object Tracking through Visual Prompting
Department of Computer Science, National Yang Ming Chiao Tung University(国立阳明交通大学计算机科学系);Research Center for Information Technology Innovation, Academia Sinica, Taiwan(学术院资讯科技创新研究中心);Microsoft, Seattle, Washington, United States(微软公司)
VCP-CLIP: A visual context prompting model for zero-shot anomaly segmentation
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);CASI Vision Technology CO., LTD.(中智视觉科技有限公司);University of Technology Sydney(悉尼科技大学);Hangzhou Dianzi University(杭州电子科技大学);Tsinghua University(清华大学)
CORA: Adapting CLIP for Open-Vocabulary Detection with Region Prompting and Anchor Pre-Matching
The Chinese University of Hong Kong(香港中文大学);SenseTime Research(商汤科技研究院);Shanghai Jiao Tong University(上海交通大学);Qing Yuan Research Institute(清源研究院);Centre for Perceptual and Interactive Intelligence(感知与交互智能研究中心)
开放词汇检测(OVD)是一个目标检测任务,旨在检测新的类别以外的基本类别的检测器被训练的对象。最近的OVD方法依赖于大规模的视觉语言预训练模型,如CLIP,用于识别新对象。我们确定了将这些模型纳入检测器训练时需要解决的两个核心障碍:(1)将在整个图像上训练的VL模型应用于区域识别任务时发生的分布不匹配;(2)定位对象的困难看不见的类。为了克服这些障碍,我们提出了CORA,DETR风格的框架,适应CLIP的区域提示和锚预匹配的开放词汇检测。区域提示通过提示基于CLIP的区域分类器的区域特征来缓解整体到区域的分布差距。锚点预匹配通过类感知的匹配机制帮助学习可泛化的对象定位。我们在COCO OVD基准上评估了CORA,在新类上我们实现了41.7 AP 50,即使没有采用额外的训练数据,也比以前的SOTA高出2.4 AP 50。当额外的训练数据是可用的,我们训练CORA $^+$的地面实况基础类别注释和额外的伪限界框标签计算的CORA。CORA $^+$在COCO OVD基准上达到43.1 AP 50,在LVIS OVD基准上达到28.1 box APr。
DeCLIP: Decoupled Prompting for Multi-Label Class-Incremental Learning with CLIP
School of Automation, Southeast University, China(东南大学自动化学院);University of the Chinese Academy of Sciences, China(中国科学院大学);Suzhou University of Science and Technology, China(苏州科技大学);Computer Vision Center, Spain(西班牙计算机视觉中心)
CLIP-DR: Textual Knowledge-Guided Diabetic Retinopathy Grading with Ranking-aware Prompting
University of Exeter(埃克塞特大学);University of Liverpool(利物浦大学);Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences(中国科学院宁波材料技术与工程研究所);Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所)
CLIP-AUTT: Test-Time Personalization with Action Unit Prompting for Fine-Grained Video Emotion Recognition
LIVIA, ILLS, Dept. of Systems Engineering, ETS Montreal(LIVIA, ILLS, 系统工程学院, 蒙特利尔高等技术学院);LIVIA, Dept. of Software and IT Engineering, ETS Montreal(LIVIA, 软件与IT工程学院, 蒙特利尔高等技术学院);Dept. of Computer Science, École Polytechnique(计算机科学系, 巴黎综合理工学院)
PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning
Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);City University of Hong Kong(香港城市大学);Yale University(耶鲁大学);Peking University(北京大学);The Chinese University of Hong Kong(香港中文大学)
Glance and Focus: Memory Prompting for Multi-Event Video Question Answering
Key Laboratory of Intelligent Information Processing of Chinese Academy of Sciences (CAS)(中国科学院智能信息处理重点实验室);Institute of Computing Technology, CAS(中国科学院计算技术研究所);University of Chinese Academy of Sciences(中国科学院大学)
End-to-end Open-vocabulary Video Visual Relationship Detection using Multi-modal Prompting
Beijing Institute of Technology(北京理工大学);School of Computer Science and Technology, Beijing Institute of Technology(北京理工大学计算机科学与技术学院);Shenzhen MSU-BIT University(深圳北理莫斯科大学);University of Rochester(罗切斯特大学)
Test-Time Multimodal Backdoor Detection by Contrastive Prompting
Chongqing University(重庆大学);Nanyang Technological University(南洋理工大学);Penn State University(宾夕法尼亚州立大学);University of Melbourne(墨尔本大学);Southeast University(东南大学)
SenCLIP: Enhancing zero-shot land-use mapping for Sentinel-2 with ground-level prompting
Mediterranean Agronomic Institute of Montpellier - CIHEAM-IAMM(地中海农业研究院-CIHEAM-IAMM);Inria(法国国家信息与自动化技术研究院);INRAE(法国国家农业研究咨询中心);Cirad(国际热带农业研究中心);UMR TETIS(TETIS联合研究单位);Univ. of Montpellier(蒙彼利埃大学)
Beyond Templates: Revisiting Zero-Shot Remote Sensing through Meta-Prompting
Remote Sensing Lab, National Technical University of Athens(遥感实验室,国家技术大学雅典分校);Remote Sensing Lab, Department of Engineering and Sciences, National Technical University of Athens(遥感实验室,工程与科学系,国家技术大学雅典分校);Universitas Mercatorum(默卡托大学)
Self-regulating Prompts: Foundational Model Adaptation without Forgetting
Mohamed bin Zayed University of AI(穆罕默德·本·扎耶德人工智能大学);Australian National University(澳大利亚国立大学);University of California, Merced(加州大学默塞德分校);Google Research(谷歌研究院);Linköping University(林雪平大学)