Contrastive Language-Image Pre-training (CLIP) has achieved excellent performance over a wide range of tasks. However, the effectiveness of CLIP heavily relies on a substantial corpus of pre-training data, resulting in notable consumption of computational resources. Although knowledge distillation has been widely applied in single modality models, how to efficiently expand knowledge distillation to vision-language foundation models with extensive data remains relatively unexplored. In this paper, we introduce CLIP-CID, a novel distillation mechanism that effectively transfers knowledge from a large vision-language foundation model to a smaller model. We initially propose a simple but efficient image semantic balance method to reduce transfer learning bias and improve distillation efficiency. This method filters out 43.7% of image-text pairs from the LAION400M while maintaining superior performance. After that, we leverage cluster-instance discrimination to facilitate knowledge transfer from the teacher model to the student model, thereby empowering the student model to acquire a holistic semantic comprehension of the pre-training data. Experimental results demonstrate that CLIP-CID achieves state-of-the-art performance on various downstream tasks including linear probe and zero-shot classification.
Modern image captioning models are usually trained with text similarity objectives. However, since reference captions in public datasets often describe the most salient common objects, models trained with text similarity objectives tend to ignore specific and detailed aspects of an image that distinguish it from others. Toward more descriptive and distinctive caption generation, we propose using CLIP, a multimodal encoder trained on huge image-text pairs from web, to calculate multimodal similarity and use it as a reward function. We also propose a simple finetuning strategy of the CLIP text e…
Kandian Content AI Lab, Platform and Content Group, Tencent（腾讯平台与内容集团看点内容AI实验室）；Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
This report describes our solution to the VALUE Challenge 2021 in the captioning task. Our solution, named CLIP4Caption++, is built on X-Linear/X-Transformer, which is an advanced model with encoder-decoder architecture. We make the following improvements on the proposed CLIP4Caption++: We employ an advanced encoder-decoder model architecture X-Transformer as our main framework and make the following improvements: 1) we utilize three strong pre-trained CLIP models to extract the text-related appearance visual features. 2) we adopt the TSN sampling strategy for data enhancement. 3) we involve t…
Linköping University（林雪平大学）；Statens Medieråd（瑞典媒体委员会）；Luleå University of Technology（吕勒奥理工大学）
Automatically identifying harmful content in video is an important task with a wide range of applications. However, there is a lack of professionally labeled open datasets available. In this work VidHarm, an open dataset of 3589 video clips from film trailers annotated by professionals, is presented. An analysis of the dataset is performed, revealing among other things the relation between clip and trailer level annotations. Audiovisual models are trained on the dataset and an in-depth study of modeling choices conducted. The results show that performance is greatly improved by combining the v…
Ultrasound-CLIP: Semantic-Aware Contrastive Pre-training for Ultrasound Image-Text Understanding
Hangzhou City University（杭州城市大学）；Hong Kong Baptist University（香港浸会大学）；Zhejiang University（浙江大学）；The First Affiliated Hospital, Zhejiang University School of Medicine（浙江大学医学院附属第一医院）；City University of Hong Kong（香港城市大学）
Facebook AI Research（脸书人工智能研究院）；Columbia University（哥伦比亚大学）；MIT CSAIL（麻省理工学院计算机科学与人工智能实验室）；ITS, CUNY Graduate Center（纽约市立大学研究生中心智能交通系统实验室）
Self-supervised learning has recently begun to rival supervised learning on computer vision tasks. Many of the recent approaches have been based on contrastive instance discrimination (CID), in which the network is trained to recognize two augmented versions of the same instance (a query and positive) while discriminating against a pool of other instances (negatives). The learned representation is then used on downstream tasks such as image classification. Using methodology from MoCo v2 (Chen et al., 2020), we divided negatives by their difficulty for a given query and studied which difficulty…
RodEpil: A Video Dataset of Laboratory Rodents for Seizure Detection and Benchmark Evaluation
BraiNE Unit, Department of Cancer Research, Luxembourg Institute of Health（卢森堡健康研究所癌症研究部门BraiNE单位）；Bioinformatics and AI Unit, Department of Medical Informatics, Luxembourg Institute of Health（卢森堡健康研究所医学信息学部门生物信息学与人工智能单位）；Multiomics Data Science Research Group, Department of Cancer Research, Luxembourg Institute of Health（卢森堡健康研究所癌症研究部门多组学数据科学研究组）；Department of Radiology, Stanford University（斯坦福大学放射科）
GR4CIL: Gap-compensated Routing for CLIP-based Class Incremental Learning
Department of COMP The Hong Kong Polytechnic University（计算机与信息科学系，香港理工大学）；Department of COMP/LSGI The Hong Kong Polytechnic University（计算机与信息科学系/LSGI，香港理工大学）
Investigating Adversarial Robustness of Multi-modal Large Language Models
Mohamed Bin Zayed University of AI, UAE（穆罕默德·本·扎耶德人工智能大学，阿联酋）；Khalifa University, UAE（哈利法大学，阿联酋）；Australian National University, Australia（澳大利亚国立大学，澳大利亚）
AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIP
School of Biomedical Engineering, Division of Life Sciences and Medicine, USTC（中国科学技术大学生命科学与医学部生物医学工程学院）；MIRACLE Center, Suzhou Institute for Advance Research, USTC（中国科学技术大学苏州高等研究院MIRACLE中心）；Key Laboratory of Intelligent Information Processing of CAS, ICT, CAS（中国科学院计算技术研究所中国科学院智能信息处理重点实验室）；State Key Laboratory of Precision and Intelligent Chemistry, USTC（中国科学技术大学精准与智能化学国家重点实验室）；Stanford University（斯坦福大学）
Koo-Fu CLIP: Closed-Form Adaptation of Vision-Language Models via Fukunaga-Koontz Linear Discriminant Analysis
Visual Recognition Group（视觉识别组）；Department of Cybernetics（cybernetics系）；Faculty of Electrical Engineering（电气工程系）；Czech Technical University in Prague（布拉格捷克技术大学）
Recent years have witnessed the rapid progress of image captioning. However, the demands for large memory storage and heavy computational burden prevent these captioning models from being deployed on mobile devices. The main obstacles lie in the heavyweight visual feature extractors (i.e., object detectors) and complicated cross-modal fusion networks. To this end, we propose LightCap, a lightweight image captioner for resource-limited devices. The core design is built on the recent CLIP model for efficient image captioning. To be specific, on the one hand, we leverage the CLIP model to extract…
Google Research（谷歌研究院）；Tel Aviv University（特拉维夫大学）；Weizmann Institute of Science（魏茨曼科学研究所）
Large vision-language models (VLMs), such as CLIP, learn rich joint image-text representations, facilitating advances in numerous downstream tasks, including zero-shot classification and text-to-image generation. Nevertheless, existing VLMs exhibit a prominent well-documented limitation - they fail to encapsulate compositional concepts such as counting. We introduce a simple yet effective method to improve the quantitative understanding of VLMs, while maintaining their overall performance on common benchmarks. Specifically, we propose a new counting-contrastive loss used to finetune a pre-trai…
NS-Net: Decoupling CLIP Semantic Information through NULL-Space for Generalizable AI-Generated Image Detection
Engineering Research Center of Digital Forensics, Ministry of Education, Nanjing University of Information Science and Technology（数字取证工程研究中心，教育部，南京信息科技大学）；University of Macau（澳门大学）
JD AI Research（京东人工智能研究院）；School of Computing and Information Systems, Singapore Management University（新加坡管理大学计算与信息系统学院）
It is not trivial to optimally learn a 3D Convolutional Neural Networks (3D ConvNets) due to high complexity and various options of the training scheme. The most common hand-tuning process starts from learning 3D ConvNets using short video clips and then is followed by learning long-term temporal dependency using lengthy clips, while gradually decaying the learning rate from high to low as training progresses. The fact that such process comes along with several heuristic settings motivates the study to seek an optimal "path" to automate the entire training. In this paper, we decompose the path…
ProtoCLIP: Prototypical Contrastive Language Image Pretraining
College of Computer and Information, Hohai University（河海大学计算机与信息学院）；Science and Technology on Underwater Vehicle Technology Laboratory（水下航行器技术科技重点实验室）；MEGVII Research（旷视科技研究院）；Nanjing Research Institute of Electronic Engineering（南京电子工程研究所）；School of EECS, Peking University（北京大学电子工程与计算机科学学院）
Columbia University（哥伦比亚大学）；Google Research（谷歌研究院）；UCLA（加州大学洛杉矶分校）
The field of vision and language has witnessed a proliferation of pre-trained foundation models. Most existing methods are independently pre-trained with contrastive objective like CLIP, image-to-text generative objective like PaLI, or text-to-image generative objective like Parti. However, the three objectives can be pre-trained on the same data, image-text pairs, and intuitively they complement each other as contrasting provides global alignment capacity and generation grants fine-grained understanding. In this work, we present a Contrastive Bi-directional Image-Text generation model (CoBIT)…
DC-Scene: Data-Centric Learning for 3D Scene Understanding
Shanghai University of Engineering Science（上海工程科学大学）；The Australian National University（澳大利亚国立大学）；Sun Yat-sen University（中山大学）；La Trobe University（拉筹伯大学）
Synthesizing Near-Boundary OOD Samples for Out-of-Distribution Detection
College of Intelligent Robotics and Advanced Manufacturing, Fudan University, Shanghai（智能机器人与先进制造学院，复旦大学，上海）；Shanghai Key Lab of Intelligent Information Processing, College of Computer Science and Artificial Intelligence, Fudan University, Shanghai（上海智能信息处理重点实验室，计算机科学与人工智能学院，复旦大学，上海）；JIIOV Technology, Beijing（JIIOV技术，北京）
Significant progress has been made on visual captioning, largely relying on pre-trained features and later fixed object detectors that serve as rich inputs to auto-regressive models. A key limitation of such methods, however, is that the output of the model is conditioned only on the object detector's outputs. The assumption that such outputs can represent all necessary information is unrealistic, especially when the detector is transferred across datasets. In this work, we reason about the graphical model induced by this assumption, and propose to add an auxiliary input to represent missing i…
