There is a growing interest in learning data representations that work well for many different types of problems and data. In this paper, we look in particular at the task of learning a single visual representation that can be successfully utilized in the analysis of very different types of images, from dog breeds to stop signs and digits. Inspired by recent work on learning networks that predict the parameters of another, we develop a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains. Our method achieves a high degree of parameter sharing while maintaining or even improving the accuracy of domain-specific representations. We also introduce the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very different visual domains and measures their ability to recognize well uniformly.
Nanyang Technological University, Singapore（新加坡南洋理工大学）；DSO National Laboratories, Singapore（新加坡国防科技研究院）
Detecting deception by human behaviors is vital in many fields such as custom security and multimedia anti-fraud. Recently, audio-visual deception detection attracts more attention due to its better performance than using only a single modality. However, in real-world multi-modal settings, the integrity of data can be an issue (e.g., sometimes only partial modalities are available). The missing modality might lead to a decrease in performance, but the model still learns the features of the missed modality. In this paper, to further improve the performance and overcome the missing modality prob…
Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；The Chinese University of Hong Kong（香港中文大学）
Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations by using large-scale contrastive image-text pairs. It shows impressive performance on zero-shot knowledge transfer to downstream tasks. To further enhance CLIP's few-shot capability, CLIP-Adapter proposed to fine-tune a lightweight residual feature adapter and significantly improves the performance for few-shot classification. However, such a process still needs extra training and computational resources. In this paper, we propose \textbf{T}raining-Free CL\textbf{IP}-\textbf…
ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation
National Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；The Chinese University of Hong Kong（香港中文大学）；Hong Kong University of Science and Technology（香港科技大学）；School of Computer Science Peking University（北京大学计算机学院）；AIRobotics
Multi-Kernel Gated Decoder Adapters for Robust Multi-Task Thyroid Ultrasound under Cross-Center Shift
Department of Astronomy and Physics, University of British Columbia（不列颠哥伦比亚大学天文与物理系）；Department of Basic and Translational Research, BC Cancer Research Institute（不列颠哥伦比亚癌症研究所在基本与转化研究部）；Department of Radiology, University of British Columbia（不列颠哥伦比亚大学放射学系）；School of Biomedical Engineering, University of British Columbia（不列颠哥伦比亚大学生物医学工程学院）
The Chinese University of Hong Kong（香港中文大学）；Shanghai AI Laboratory（上海人工智能实验室）；SenseTime Research（商汤科技研究院）；Centre for Perceptual and Interactive Intelligence（感知与智能交互中心）
Contrastive Vision-Language Pre-training, known as CLIP, has provided a new paradigm for learning visual representations using large-scale image-text pairs. It shows impressive performance on downstream tasks by zero-shot knowledge transfer. To further enhance CLIP's adaption capability, existing methods proposed to fine-tune additional learnable modules, which significantly improves the few-shot performance but introduces extra training time and computational resources. In this paper, we propose a training-free adaption method for CLIP to conduct few-shot classification, termed as Tip-Adapter…
Adapter-Enhanced Semantic Prompting for Continual Learning
Beijing Key Laboratory of Multimedia and Intelligent Software Technology（北京多媒体与智能软件技术重点实验室）；Faculty of Information Technology, Beijing University of Technology（信息科技学院，北京理工大学）；University of California at Merced（加州大学默塞德分校）；Macquarie University（麦考瑞大学）
Dyn-Adapter: Towards Disentangled Representation for Efficient Visual Recognition
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；Chinese Academy of Sciences（中国科学院）；Meituan Inc.（美团公司）；Cooperative Medianet Innovation Center (CMIC), Shanghai Jiao Tong University（上海交通大学合作 medianet 创新中心）
CLIP-Adapter: Better Vision-Language Models with Feature Adapters
Shanghai AI Laboratory（上海人工智能实验室）；CUHK MMLab（香港中文大学多媒体实验室）；Rutgers University（罗格斯大学）；Centre for Perceptual and Interactive Intelligence (CPII)（感知与交互智能中心）
GRACE: Grounded Reasoning via Adapter Composition and Evidence-Aware Calibration for Educational Visual Question Answering
Columbia University（哥伦比亚大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）；Stevens Institute of Technology（史蒂文斯理工学院）；Georgetown University（乔治城大学）；San Francisco State University（旧金山州立大学）；Loughborough University（拉夫堡大学）；Texas A&M University（德克萨斯农工大学）
DA-Ada: Learning Domain-Aware Adapter for Domain Adaptive Object Detection
Intelligent Software Research Center, Institute of Software, CAS（中国科学院软件研究所智能软件研究中心）；State Key Lab of Processors, Institute of Computing Technology, CAS（中国科学院计算技术研究所处理器国家重点实验室）；State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, CAS（中国科学院自动化研究所多模态人工智能系统国家重点实验室）；University of Chinese Academy of Sciences（中国科学院大学）
From Holistic to Localized: Local Enhanced Adapters for Efficient Visual Instruction Fine-Tuning
Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University（上海智能信息处理关键实验室，复旦大学计算机学院）；Shanghai Collaborative Innovation Center on Intelligent Visual Computing（上海智能视觉计算协同创新中心）；Singapore Management University（新加坡管理大学）
1%>100%: High-Efficiency Visual Adapter with Complex Linear Projection Optimization
BNRist, Department of Computer Science；Technology, Tsinghua University, Beijing, China；School of Automation；Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China；Nankai lnternational Advanced Research Institute (Shenzhen Futian) \& SLAI, Shenzhen, China
Nanjing University（南京大学）；Shanghai AI Laboratory（上海人工智能实验室）
In this report, we present our champion solution to the WSDM2023 Toloka Visual Question Answering (VQA) Challenge. Different from the common VQA and visual grounding (VG) tasks, this challenge involves a more complex scenario, i.e. inferring and locating the object implicitly specified by the given interrogative question. For this task, we leverage ViT-Adapter, a pre-training-free adapter network, to adapt multi-modal pre-trained Uni-Perceiver for better cross-modal localization. Our method ranks first on the leaderboard, achieving 77.5 and 76.347 IoU on public and private test sets, respectiv…
In this paper, we look at the problem of cross-domain few-shot classification that aims to learn a classifier from previously unseen classes and domains with few labeled samples. Recent approaches broadly solve this problem by parameterizing their few-shot classifiers with task-agnostic and task-specific weights where the former is typically learned on a large training set and the latter is dynamically predicted through an auxiliary network conditioned on a small support set. In this work, we focus on the estimation of the latter, and propose to learn task-specific weights from scratch directl…
