CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
Columbia University（哥伦比亚大学）；Microsoft Research（微软研究院）
Zhecan Wang, Noel Codella, Yen-Chun Chen, Luowei Zhou, Jianwei Yang, Xiyang Dai, Bin Xiao, Haoxuan You, Shih-Fu Chang, Lu Yuan
备注：
This paper is greatly modified and updated to be re-submitted to another conference. The new paper is under the name "Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks", https://doi.org/10.48550/arXiv.2204.10496
Contrastive language-image pretraining (CLIP) links vision and language modalities into a unified embedding space, yielding the tremendous potential for vision-language (VL) tasks. While early concurrent works have begun to study this potential on a subset of tasks, important questions remain: 1) What is the benefit of CLIP on unstudied VL tasks? 2) Does CLIP provide benefit in low-shot or domain-shifted scenarios? 3) Can CLIP improve existing approaches without impacting inference or pretraining complexity? In this work, we seek to answer these questions through two key contributions. First, we introduce an evaluation protocol that includes Visual Commonsense Reasoning (VCR), Visual Entailment (SNLI-VE), and Visual Question Answering (VQA), across a variety of data availability constraints and conditions of domain shift. Second, we propose an approach, named CLIP Targeted Distillation (CLIP-TD), to intelligently distill knowledge from CLIP into existing architectures using a dynamically weighted objective applied to adaptively selected tokens per instance. Experiments demonstrate that our proposed CLIP-TD leads to exceptional gains in the low-shot (up to 51.9%) and domain-shifted (up to 71.3%) conditions of VCR, while simultaneously improving performance under standard fully-supervised conditions (up to 2%), achieving state-of-art performance on VCR compared to other single models that are pretrained with image-text data only. On SNLI-VE, CLIP-TD produces significant gains in low-shot conditions (up to 6.6%) as well as fully supervised (up to 3%). On VQA, CLIP-TD provides improvement in low-shot (up to 9%), and in fully-supervised (up to 1.3%). Finally, CLIP-TD outperforms concurrent works utilizing CLIP for finetuning, as well as baseline naive distillation approaches. Code will be made available.
Hong Kong University of Science and Technology（香港科技大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）
The recent large-scale vision-language pre-training (VLP) of dual-stream architectures (e.g., CLIP) with a tremendous amount of image-text pair data, has shown its superiority on various multimodal alignment tasks. Despite its success, the resulting models are not capable of multimodal generative tasks due to the weak text encoder. To tackle this problem, we propose to augment the dual-stream VLP model with a textual pre-trained language model (PLM) via vision-language knowledge distillation (VLKD), enabling the capability for multimodal generation. VLKD is pretty data- and computation-efficie…
Unifying Biomedical Vision-Language Expertise: Towards a Generalist Foundation Model via Multi-CLIP Knowledge Distillation
Department of Radiation Oncology, Winship Cancer Institute, Emory University School of Medicine（放射肿瘤科，Winship癌症研究所，埃默里大学医学院）；Department of Biomedical Engineering, College of Engineering, Georgia Institute of Technology（生物医学工程系，工程学院，佐治亚理工学院）；Department of Computer Science and Mathematics, Laney Graduate School, Emory University（计算机科学与数学系，兰尼研究生院，埃默里大学）；School of Electrical and Computer Engineering, College of Engineering, Georgia Institute of Technology（电气与计算机工程系，工程学院，佐治亚理工学院）
Adversarial Prompt Distillation for Vision-Language Models
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（上海智能信息处理实验室，计算机科学学院，复旦大学）；The Chinese University of Hong Kong, Shatin, Hong Kong（香港中文大学，沙田，香港）；The University of Hong Kong, Pokfulam, Hong Kong（香港大学，薄扶林，香港）
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training
Technical University of Munich（慕尼黑工业大学）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；Helmholtz Munich（亥姆霍兹慕尼黑中心）；Munich Data Science Institute (MDSI)（慕尼黑数据科学研究所）
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Peng Cheng Laboratory（鹏城实验室）；University of Chinese Academy of Sciences（中国科学院大学）
Although significant progress has been made in few-shot learning, most of existing few-shot image classification methods require supervised pre-training on a large amount of samples of base classes, which limits their generalization ability in real world application. Recently, large-scale Vision-Language Pre-trained models (VLPs) have been gaining increasing attention in few-shot learning because they can provide a new paradigm for transferable visual representation learning with easily available text on the Web. However, the VLPs may neglect detailed visual information that is difficult to de…
Parameter-Efficient CLIP Adaptation for 3D Understanding via Unified Tokenization
Fondazione Bruno Kessler（布鲁诺·科塞拉基金会）；University of Trento（特伦托大学）；University of Pisa（比萨大学）；Beijing Forestry University（北京林业大学）；Shanghai Jiao Tong University（上海交通大学）；Hong Kong University of Science and Technology (GZ)（香港科技大学）；Shandong University（山东大学）；University of California, Merced（加州大学默塞德分校）
Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing
Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；National University of Singapore（新加坡国立大学）；Tencent（腾讯）；Wuhan AI Research（武汉人工智能研究院）
Dyslexify: A Mechanistic Defense Against Typographic Attacks in CLIP
Fraunhofer Heinrich Hertz Institute（弗劳恩霍夫海因里希·赫兹研究所）；University of Oxford（牛津大学）；Technological University Dublin（都柏林技术大学）；Technische Universität Berlin（柏林技术大学）
Vision-language models (VLMs) exhibit uneven performance across languages, a problem that is often exacerbated when the model size is reduced. While Knowledge distillation (KD) demonstrates promising results in transferring knowledge from larger to smaller VLMs, applying KD in multilingualism is an underexplored area. This paper presents a controlled empirical study of KD behavior across five distillation approaches, isolating their effects on cross-lingual representation consistency and downstream performance stability under model compression. We study five distillation formulations across CL…
Proxy Robustness in Vision Language Models is Effortlessly Transferable
Chongqing Key Laboratory of Bio-perception（重庆生物感知实验室）；Multimodal Intelligent Information Processing, Chongqing University, Chongqing 401331, China（多模态智能信息处理，重庆大学，重庆401331，中国）；School of Microelectronics（微电子学院）；Communication Engineering, Chongqing University, Chongqing 401331, China（通信工程，重庆大学，重庆401331，中国）；School of Data Science, Lingnan University, Hong Kong（数据科学学院，岭南大学，香港）
CLIP-guided Federated Learning on Heterogeneous and Long-Tailed Data
Institute of Artificial Intelligence（人工智能研究院）；School of Informatics, Xiamen University（厦门大学信息学院）；East China Normal University（华东师范大学）；Chongqing Institute of East China Normal University（华东师范大学重庆研究院）
UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing
Center for Data Science, Peking University（北京大学数据科学中心）；Alibaba Group（阿里巴巴集团）；CASIA；Center for Machine Learning Research, Peking University（北京大学机器学习研究中心）；State Key Laboratory of General Artificial Intelligence, Peking University（北京大学通用人工智能国家重点实验室）
Exploring Weak-to-Strong Generalization for CLIP-based Classification
School of Computing and Information Systems University of Melbourne, Australia（墨尔本大学计算机与信息系统学院）；School of Computing and Information Systems University of Melbournem, Australia（墨尔本大学计算机与信息系统学院）；School of Computer Science and Engineering Southeast University, China（东南大学计算机科学与工程学院）
