Recent advances in vision-language models (VLMs) have made significant progress in downstream tasks that require quantitative concepts such as facial age estimation and image quality assessment, enabling VLMs to explore applications like image ranking and retrieval. However, existing studies typically focus on the reasoning based on a single image and heavily depend on text prompting, limiting their ability to learn comprehensive understanding from multiple images. To address this, we propose an effective yet efficient approach that reframes the CLIP model into a learning-to-rank task and introduces a lightweight adapter to augment CLIP for text-guided image ranking. Specifically, our approach incorporates learnable prompts to adapt to new instructions for ranking purposes and an auxiliary branch with ranking-aware attention, leveraging text-conditioned visual differences for additional supervision in image ranking. Our ranking-aware adapter consistently outperforms fine-tuned CLIPs on various tasks and achieves competitive results compared to state-of-the-art models designed for specific tasks like facial age estimation and image quality assessment. Overall, our approach primarily focuses on ranking images with a single instruction, which provides a natural and generalized way of learning from visual differences across images, bypassing the need for extensive text prompts tailored to individual tasks. Code is available: github.com/uynaes/RankingAwareCLIP.
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
Shanghai AI Laboratory（上海人工智能实验室）；CUHK MMLab（香港中文大学多媒体实验室）；Rutgers University（罗格斯大学）；Centre for Perceptual and Interactive Intelligence (CPII)（感知与交互智能中心）
PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition
Univ. BTU Cottbus-Senftenberg（科特布斯-森夫滕堡应用科学大学）；Sorbonne University Abu Dhabi（索邦大学阿布扎比分校）；Univ. Polytechnique Hauts-de-France（上法兰西理工大学）；Centrale Lille（里尔中央理工学院）；CNRS（法国国家科学研究中心）
Semantic Relation-Enhanced CLIP Adapter for Domain Adaptive Zero-Shot Learning
School of Computer Science and Technology, East China Normal University, China（东华大学计算机科学与技术学院）；College of Computer Science and Technology, Ocean University of China, China（中国海洋大学计算机科学与技术学院）
Peking University（北京大学）；Microsoft Research Asia（微软亚洲研究院）
Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explor…
TDS-CLIP: Temporal Difference Side Network for Efficient VideoAction Recognition
School of Electrical and Electronic Engineering, Shandong University of Technology（山东理工大学电子与电气工程学院）；School of Control Scienceand Engineering, Shandong University（山东大学控制科学与工程学院）；School of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics（南京航空航天大学人工智能学院）；Pillar of Information Systems Technology and Design, Singapore University of Technology and Design（新加坡科技设计大学信息系统技术与设计支柱）
University College London（伦敦大学学院）；University of Edinburgh（爱丁堡大学）；Alan Turing Institute（艾伦·图灵研究所）；Niantic（Niantic公司）
Vision-language models such as CLIP are pretrained on large volumes of internet sourced image and text pairs, and have been shown to sometimes exhibit impressive zero- and low-shot image classification performance. However, due to their size, fine-tuning these models on new datasets can be prohibitively expensive, both in terms of the supervision and compute required. To combat this, a series of light-weight adaptation methods have been proposed to efficiently adapt such models when limited supervision is available. In this work, we show that while effective on internet-style datasets, even th…
A Simple Efficiency Incremental Learning Framework via Vision-Language Model with Nonlinear Multi-Adapters
Faculty of Information Technology, University of Jyväskylä（信息科技学院，于韦斯屈耶大学）；National University of Singapore（新加坡国立大学）；Lab of Brain-Machine Intelligence, Zhejiang University（脑机智能实验室，浙江大学）；School of Computer Science and Technology, Dalian University of Technology（计算机科学与技术学院，大连理工大学）
CLIP-SLA: Parameter-Efficient CLIP Adaptation for Continuous Sign Language Recognition
King Fahd University of Petroleum and Minerals（法赫德石油和矿产大学）；Imam Abdulrahman Bin Faisal University（伊玛目阿卜杜勒拉赫曼·本·费萨尔大学）；SDAIA–KFUPM Joint Research Center for Artificial Intelligence（SDAIA-KFUPM人工智能联合研究中心）
Test-time Alignment-Enhanced Adapter for Vision-Language Models
school of computer science；engineering sun yat-sen university Guangzhou, China；school of artificial intelligence sun yat-sen university Guangzhou, China
Graph-guided Cross-composition Feature Disentanglement for Compositional Zero-shot Learning
PowerChina Huadong Engineering Corporation Limited（国家电力投资集团华东工程公司）；Hangzhou Dianzi University（杭州电子科技大学）；The University of Manchester（曼彻斯特大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Zhejiang University（浙江大学）
Open-Vocabulary X-ray Prohibited Item Detection via Fine-tuning CLIP
College of Information Science and Engineering, Northeastern University（东北大学信息科学与工程学院）；Foshan Graduate School of Innovation, Northeastern University（东北大学佛山创新研究院）
针对 X 射线安检中新增违禁品类别识别难题，本文提出带瓶颈适配器的 OVXD 模型，通过微调 CLIP 对齐 X 射线图像与文本，并在 PIXray、PIDray 上显著提升新类别检测性能。
Text-promptable Object Counting via Quantity Awareness Enhancement
College of Electronic and Information Engineering, Tongji University（同济大学电子与信息学院）；College of Computer Science, Wuhan University（武汉大学计算机学院）；College of Computer Science, Tongji University（同济大学计算机学院）；State Key Laboratory of Autonomous Intelligent Unmanned Systems（自主智能无人系统国家重点实验室）
AnchorScore: A CLIP-Based Diagnostic of MLLM Annotation Difficulty
School of Foreign Studies, Changsha University of Science and Technology（长沙理工大学外国语学院）；School of Education, Hunan Agricultural University（湖南农业大学教育学院）；School of Information and Intelligence, Hunan Agricultural University（湖南农业大学信息与智能科学学院）
Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）
Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations by using large-scale contrastive image-text pairs. It shows impressive performance on zero-shot knowledge transfer to downstream tasks. To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification. However, such a process still needs extra training and computational resources. In this paper, we propose \textbf{T}raining-Free CL\textbf{IP}-\textbf…
HFIPS, Chinese Academy of Sciences（中国科学院合肥物质科学研究院）；University of Science and Technology of China（中国科学技术大学）；University of Chicago（芝加哥大学）；Carnegie Mellon University（卡内基梅隆大学）
Labeling large image datasets with attributes such as facial age or object type is tedious and sometimes infeasible. Supervised machine learning methods provide a highly accurate solution, but require manual labels which are often unavailable. Zero-shot models (e.g., CLIP) do not require manual labels but are not as accurate as supervised ones, particularly when the attribute is numeric. We propose a new approach, CLIPPR (CLIP with Priors), which adapts zero-shot models for regression and classification on unlabelled datasets. Our method does not use any annotated images. Instead, we assume a …
Canadian Institute for Advanced Research（加拿大高等研究院）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；University of Guelph（圭尔夫大学）；Apple Inc（苹果公司）
Deep neural networks require collecting and annotating large amounts of data to train successfully. In order to alleviate the annotation bottleneck, we propose a novel self-supervised representation learning approach for spatiotemporal features extracted from videos. We introduce Skip-Clip, a method that utilizes temporal coherence in videos, by training a deep model for future clip order ranking conditioned on a context clip as a surrogate objective for video future prediction. We show that features learned using our method are generalizable and transfer strongly to downstream tasks. For acti…
